ClearFoundation

Hi Everyone,
I noticed today that the Remote Server backup sent me an email, stated a error occurred.

Control Socket read error, any idea?

Been down for over two days now...

If you look in your logs you will probably see these errors...

www.clearfoundation.com/component/option...limitstart,10/#15717

Hi,

I can confirm that the error is still present.

David

Same here - tried two machines - both failed...

Seems like the ClearOS team are just not interested in getting it working properly. No response to this thread, or in the other here... www.clearfoundation.com/component/option...view/id,15825/#15825

It's not lack of interest Tony...if Darryl (employed with ClearCenter and 'owner/coder' of this particular module/service) would like to post details to the community when he comes back to work he can, otherwise, for sake of privacy, all I can say is that he had to suddenly take some time off to be with friends and family.

B.

Yeah I figured something was up, not like ClearOS to not be interested... No biggie really, if I knew the guts of it I would help out... programming not my specialty, hacking yes, programming no.

P.S. I still think 5.2 is the best build so far. I can't wait until 6.0...

If Darryl has suffered an illness, death or other calamity within family and or friends, than I am truly sorry to hear that.

However Ben, this module has not just suddenly developed a problem in the last week or two after having been rock solid from introduction until now. Continual difficulties go back over many months virtually right to the time it was first introduced...

Hi Tony,

That's a fair comment Tony, and I won't dispute the fact that there's been problems.

Darryl can explain in more detail, of course, but the problems over the last couple of months have been with the number of devices connecting back to the SDN and the resource management on the storage side.

As I understand (and saw), when we deployed the service, it was working well...when hundreds or thousands of systems start trying to sync daily, that's when problems were discovered. Obviously, this type of scalability needs to be built in and we're working towards it. However, I know many of you will appreciate the number of hours that can potentially go into re-writing even a portion of a program's logic as complicated as remote backup. It might seem a simple enough concept, but there are many technologies in place, mainly to keep your (collectively) data absolutely secure.

What's displayed to the end user (i.e. 'socket error') cannot adequately describe what code changes might be necessary...and that's why, aside from helping get 5.2 out the door, the remote backup has been Darryl's primary focus for the quarter.

Believe me...we care!

Ben

Thank-you Ben. That's the sort of feedback that shows us ClearOS does really care and is much appreciated! It also illustrates the importance of communication... people do not like being treated as mushrooms

Tony,

I have isolated three remote backup issues and should have a set of patches (and RPM upgrades) in short order. The first issue was an authentication problem with our SDN secure server where the database connection would not reconnect properly if lost. This has been resolved as about an hour ago. The second issue is a scalability issue with the iSCSI Enterprise Target software we use. It seems there is a known bug which prohibits initiators from connecting after there is 97 targets exported. The solution to this may take more time. The long-term solutions are running multiple VMs to share the load, patching the IET daemon ourselves, or wait for an upstream patch. The short-term solution will be to push out a an RPM update which switches the back-up mode to use a more traditional rsync method eliminating the use of iSCSI.

Finally the third issue is a connection drop/resume bug that can occur with large file-systems when using the traditional rsync method. I am still in the process of studying this issue and have yet to pin down the exact details.

The three error codes that clients will see from the above problems are:

[ServiceException::CODE_ISCSI_DISCOVERY] ServiceException
This is returned when the IET daemon has exceed 96 exports.

[ControlSocketException::CODE_READ] ControlSocketException
This is returned when the secure server could not authenticate because the database connection is defunct. This has now been resolved. However, this error is rather generic so it can be returned for other read errors.

[ProtocolException::CODE_ERROR] 900:ProtocolException::CODE_UNEXPECTED
This is usually returned when performing an audit of a large file-system using the traditional rsync method. It can be returned rarely for other reasons too.

I'll continue to regularly post my progress through these issue here...