ClearFoundation

Hi All,

Hope someone can shed some light on this:

I have 'inherited' a client who has 2 x ClearOS 5.2 boxes, server 1 - Stand alone, no firewall (used as PDC & file server with flexshares), server 2 - Gateway mode & DHCP. Client PC's are a mix of Win XP SP3 & Win 7.

Problem is, everyday @ 3pm client PC's can't access the flexshares (network path not found)? this lasts for +/- 30min (some times longer) and then all is well again until the next day same time. Strangely, while it is not accessible, ping requests return with >3ms from client PC's.

ifconfig:

eth0 Link encap:Ethernet HWaddr 70:71:BC:4F:3D:D6
inet addr:111.111.111.1 Bcast:111.111.111.255 Mask:255.255.255.0
inet6 addr: fe80::7271:bcff:fe4f:3dd6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:43390055 errors:0 dropped:1981 overruns:0 frame:0
TX packets:16692047 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4023963381 (3.7 GiB) TX bytes:1475309367 (1.3 GiB)
Memory:fbbc0000-fbbe0000

iperf -c :

Client connecting to 111.111.111.1, TCP port 5001
TCP window size: 8.00 KBytes (default)

[128] local 111.111.111.160 port 52991 connected with 111.111.111.1 port 5001
[ ID] Interval Transfer Bandwidth
[128] 0.0-10.0 sec 377 MBytes 316 Mbits/sec

No other process are running at this point (as far as I can tell)

Don't have a lot of experience with linux. Not even sure where to start looking or what I'm looking for!

Hi Raymond, any clues in the log files around this time? /var/log/messages or /var/log/cron (which handles scheduled tasks) or indeed /var/log/system

Following up on Tim's post, it may be worth looking for the key word "run-parts" to see if it is happening when a major "nightly" cron job is running. Is your system clock/timezone set correctly because these nightly (and weekly and monthly) jobs should happen at around 4am?

Hi Tim,

Thanks for your reply!!

/var/log/messages:

Jun 28 14:59:21 setsquared-01 smbd[25965]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 14:59:43 setsquared-01 smbd[16349]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 15:10:03 setsquared-01 smbd[25965]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 15:34:09 setsquared-01 smbd[5080]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 15:44:10 setsquared-01 smbd[5080]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 15:54:12 setsquared-01 smbd[5080]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 16:04:13 setsquared-01 smbd[5080]: nss_ldap: could not search LDAP server - Server is unavailable
Jun 28 16:24:16 setsquared-01 smbd[5080]: nss_ldap: could not search LDAP server - Server is unavailable

These logs show throughout the messages log file.

/var/log/cron:

Jun 28 15:01:01 setsquared-01 crond[9933]: (root) CMD (run-parts /etc/cron.hourly)
Jun 28 15:01:01 setsquared-01 crond[9934]: (root) CMD (/usr/sbin/log2mysql >/dev/null 2>&1)
Jun 28 15:05:01 setsquared-01 crond[12585]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:05:01 setsquared-01 crond[12586]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:05:01 setsquared-01 crond[12587]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:05:01 setsquared-01 crond[12589]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:05:01 setsquared-01 crond[12596]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:10:01 setsquared-01 crond[17221]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:10:01 setsquared-01 crond[17223]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:10:01 setsquared-01 crond[17225]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:10:01 setsquared-01 crond[17226]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:10:01 setsquared-01 crond[17233]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:15:01 setsquared-01 crond[23333]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:15:01 setsquared-01 crond[23332]: (root) CMD (/var/webconfig/scripts/archive_mesg.php >/dev/null 2>&1)
Jun 28 15:15:01 setsquared-01 crond[23335]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:15:01 setsquared-01 crond[23334]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:15:01 setsquared-01 crond[23336]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:15:01 setsquared-01 crond[23337]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:20:01 setsquared-01 crond[30732]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:20:01 setsquared-01 crond[30731]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:20:01 setsquared-01 crond[30733]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:20:01 setsquared-01 crond[30737]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:20:01 setsquared-01 crond[30741]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:25:01 setsquared-01 crond[6425]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:25:01 setsquared-01 crond[6426]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:25:01 setsquared-01 crond[6429]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:25:01 setsquared-01 crond[6431]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:25:01 setsquared-01 crond[6432]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:27:01 setsquared-01 crond[9640]: (root) CMD ( /usr/sbin/pflogsumm -d today /var/log/maillo*[g1] > /var/webconfig/reports/postfix/data-today.out)
Jun 28 15:30:01 setsquared-01 crond[13944]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)J
Jun 28 15:30:01 setsquared-01 crond[13945]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:30:01 setsquared-01 crond[13948]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:30:01 setsquared-01 crond[13950]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:30:01 setsquared-01 crond[13954]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:30:01 setsquared-01 crond[13955]: (root) CMD (/var/webconfig/scripts/archive_mesg.php >/dev/null 2>&1)
Jun 28 15:35:01 setsquared-01 crond[21314]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:35:01 setsquared-01 crond[21316]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:35:01 setsquared-01 crond[21318]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:35:01 setsquared-01 crond[21322]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:35:01 setsquared-01 crond[21327]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:40:01 setsquared-01 crond[28635]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:40:01 setsquared-01 crond[28637]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:40:01 setsquared-01 crond[28638]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:40:01 setsquared-01 crond[28641]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:40:01 setsquared-01 crond[28643]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:45:01 setsquared-01 crond[3693]: (root) CMD (/var/webconfig/scripts/archive_mesg.php >/dev/null 2>&1)
Jun 28 15:45:01 setsquared-01 crond[3694]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:45:01 setsquared-01 crond[3695]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:45:01 setsquared-01 crond[3699]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:45:01 setsquared-01 crond[3703]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:45:01 setsquared-01 crond[3706]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:50:01 setsquared-01 crond[11576]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 15:50:01 setsquared-01 crond[11577]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:50:01 setsquared-01 crond[11578]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:50:01 setsquared-01 crond[11580]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:50:01 setsquared-01 crond[11586]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:55:01 setsquared-01 crond[19703]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 15:55:01 setsquared-01 crond[19705]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 15:55:01 setsquared-01 crond[19706]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 15:55:01 setsquared-01 crond[19712]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 15:55:01 setsquared-01 crond[19714]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 16:00:01 setsquared-01 crond[26190]: (root) CMD (/var/webconfig/scripts/archive_mesg.php >/dev/null 2>&1)
Jun 28 16:00:01 setsquared-01 crond[26191]: (root) CMD (/var/flexshare/scripts/check_mbox.php >/dev/null 2>&1)
Jun 28 16:00:01 setsquared-01 crond[26193]: (root) CMD (/etc/mrtg/mrtg_run; LANG=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg >/dev/null 2>/dev/null)
Jun 28 16:00:01 setsquared-01 crond[26195]: (root) CMD (LANG=en_US /usr/share/system/scripts/servicewatch >/dev/null)
Jun 28 16:00:01 setsquared-01 crond[26196]: (root) CMD (/var/webconfig/scripts/archive_mbox.php >/dev/null 2>&1)
Jun 28 16:00:01 setsquared-01 crond[26205]: (root) CMD (/usr/sbin/filtertraining >/dev/null 2>&1)
Jun 28 16:01:01 setsquared-01 crond[26385]: (root) CMD (run-parts /etc/cron.hourly)

I see that the "run-parts" as mentioned by Nick Howitt is running on an hourly basis, is this behavior normal?

/var/log/system:

Jun 28 09:45:43 setsquared-01 webconfig: login - root login successful
Jun 28 10:31:45 setsquared-01 cc-firewall: Starting firewall...
Jun 28 10:31:45 setsquared-01 cc-firewall: Loading environment
Jun 28 10:31:45 setsquared-01 cc-firewall: Detected WAN role for interface: eth0
Jun 28 10:31:45 setsquared-01 cc-firewall: Detected WAN info - eth0 111.111.111.1 on network 111.111.111.0/24
Jun 28 10:31:45 setsquared-01 cc-firewall: Setting kernel parameters
Jun 28 10:31:45 setsquared-01 cc-firewall: Using trusted standalone mode (no firewall)
Jun 28 10:31:45 setsquared-01 cc-firewall: Unloading unecessary NAT kernel modules
Jun 28 10:31:46 setsquared-01 cc-firewall: Loading kernel modules
Jun 28 10:31:46 setsquared-01 cc-firewall: Setting default policy to ACCEPT
Jun 28 10:31:46 setsquared-01 cc-firewall: Defining custom chains
Jun 28 10:31:46 setsquared-01 cc-firewall: Running custom rules
Jun 28 10:31:46 setsquared-01 cc-firewall: Running user-defined proxy rules
Jun 28 10:31:46 setsquared-01 cc-firewall: Execution time: 0.695s
Jun 28 10:31:46 setsquared-01 firewall: ========== start /etc/rc.d/rc.firewall.local ==========
Jun 28 10:31:46 setsquared-01 firewall: # Custom firewall rules.
Jun 28 10:31:46 setsquared-01 firewall: # This file is executed by the firewall on stop/start/restart.
Jun 28 10:31:46 setsquared-01 firewall: ========== start /etc/rc.d/rc.firewall.local ==========
Jun 28 14:16:01 setsquared-01 webconfig: login - root login successful
Jun 29 03:31:01 setsquared-01 software-update: starting
Jun 29 03:31:01 setsquared-01 software-update: checking for recommended updates
Jun 29 03:31:18 setsquared-01 software-update: done
Jun 29 07:32:03 setsquared-01 service-update: antimalware up-to-date
Jun 29 07:32:06 setsquared-01 service-update: content filter up-to-date
Jun 29 08:11:07 setsquared-01 webconfig: login - root login successful

The only thing that looks out of place is the LDAP errors, or am I way off the mark?

Thanks again!
Ray

Update:
Spent Friday going through the entire network, top to bottom:

All cabling, switches, routers, wireless access points were replaced October 2010 and are in good working condition.
Servers themselves were replaced at the same time too.

I read a post by Tomas (Standalone to gateway - slow and packets dropped: www.clearfoundation.com/component/option.../func,view/id,33784/) and did notice that BOTH Server's disk drives transfer speeds were poor as they are SATA but not set to run AHCI in the BIOS. I have amended this - thanks Tony Ellis + Nick Howitt !!!

Now, wait and see what happens today @ 3pm...

Hi Nick,

Thank you for your reply!!

Date & time zone settings are correct through webconfig: Jun 29 2012 09:07:08 SAST

This is the only reference I can find to the "run-parts":

Jun 28 16:01:01 setsquared-01 crond[26385]: (root) CMD (run-parts /etc/cron.hourly)

According to /var/log/system software updates are taking place @ 03:30am with completion @ 07:30am.

Thanks again for your input!

Ray

I forgot about that one. run-parts will run hourly for anything in the /etc/cron.hourly folder. If there was anything in there at fault you'd have a problem every hour which is not the case. If your date and timezone is correct that blows my idea out.

HI Nick,

No worries, thanks for the input.

Update:

Hi All,

After some pain-staking troubleshooting, i found the problem child!

Cobian backup v10 was running 2 workstation backups at precisely 3pm everyday

Strange that this would consume all the bandwidth to the file server...

As this is the cause, what would be a recommended backup software?

Thanks,
Ray

If you are trying to back up PC's try BackupPC.

Thanks Nick!

Will give it a go!