1. Store
  2. Apps
  3. Hardware
  4. Support
  5. Solutions

ClearFoundation

Forums
Welcome, Guest
Zarafa Webmail - Spam Training / Bayesian Learning
(1 viewing) 1 Guest
Go to bottomPage: 1234
TOPIC: Zarafa Webmail - Spam Training / Bayesian Learning
*
#23259
Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 7 Months ago  
This how to will configure a public folder for spam so that Zarafa users can "train" the SpamAssassin spam filter to be more accurate. This folder is accessible via the Webmail or Outlook. I assume you already have Zarafa installed and running as per the forum thread - you will also need the Zarafa-Gateway service running

Big thanks to the SME thread here, the steps below are derived from it
wiki.contribs.org/Zarafa_Bayesian_Learning

  1. Configure ClearOS LDAP to use Zarafa LDAP schema
    Code:

    cp /usr/share/zarafa/zarafa.schema /etc/openldap/schema/
    vi /etc/openldap/slapd.conf
    ## add the following line after the top section
    include /etc/openldap/schema/zarafa.schema
    ## restart LDAP
    service ldap restart



  2. Modify webconfig php so that it recognises the Zarafa LDAP parameters
    Edit /var/webconfig/api/User.class.php, add the "zarafa-user" to the core class list. Around line 210:
    Code:

    $this->coreclasses = array(
            'top',
            'posixAccount',
            'shadowAccount',
            'inetOrgPerson',
            'kolabInetOrgPerson',
            'hordePerson',
            'pcnAccount',
            'zarafa-user',
    );



  3. Create the Zarafa Public store (if you haven't already)
    Code:

    zarafa-admin -s



  4. Create a user account via the webconfig with mail permissions only specifically for collecting the mail. I've used 'SpamAdmin' account below


  5. Use LdapAdmin or similar tool to assign a user Zarafa admin account, and add zarafa-user to object class. See screenshots below.
    a. First set your LDAP publish policy to enabled within the webconfig
    b. Use LdapAdmin to login using your LDAP credentials Bind DN, Base, and Bind Password
    c. Right click on your user, and add the Zarafa-user object class to the drop down list on the left hand side
    d. Then change the Zarafa-Admin attribute to 1


  6. Create ham/spam folders and assign permissions
    Login to Zarafa with an account that has admin rights as assigned above and make two new folders LearnAsSpam and LearnAsHam under: Public folder > Public folders. Set the permissions (right-click folder > Properties > Permission-tab) on both these new folders to:

    Spam administration account
    * Folder visible
    * Read items
    * Edit items: all
    * Delete items: all

    Everyone (and/or other users/groups you've added at least need:)
    * Folder visible
    * Create items
    * Edit items: none
    * Delete items: none


  7. Create a script that will login via IMAP check the folders periodically and pull mail into the ClearOS spam training system. Note that you will need the Zarafa Gateway service running for IMAP access. You can run this on a seperate port if you are still using the ClearOS IMAP. I've assumed that Zarafa Gateway is listening on port 143.

    Create /usr/bin/Zarafa-sa-learn.pl
    Code:

    #!/usr/bin/perl
    #
    # Extract mail from imap server shared folder 'Public folders/LearnAsSpam' & 'Public folders/LearnAsHam'
    # Orig by dmz@dmzs.com - March 19, 2004
    # http://www.dmzs.com/tools/files/spam.phtml
    # Modified for compatibility with ClearOS spam training Jan 14, 2011, spam training handled by external script
    # LGPL

    use Mail::IMAPClient;

    my $debug=0;
    my $salearn;

    my $filterdir="/var/spool/filter/training";
    my $spamwebclient="$filterdir/spam-web";
    my $notspamwebclient="$filterdir/notspam-web";

    # # # # # # # # # # EDIT USER AND PASSWORD (CHECK PORT) # # # # # # # # # #
    my $imap = Mail::IMAPClient->new( Server=> '127.0.0.1:143',
                                      User => 'spamadmin',
                                      Password => 'yourpasswordhere',
                                      Debug => $debug);

    if (!defined($imap)) { die "IMAP Login Failed"; }

    # If debugging, print out the total counts for each mailbox
    if ($debug) {
             my $spamcount = $imap->message_count('Public folders/LearnAsSpam');
             print $spamcount, " Spam to process\n";

             my $nonspamcount = $imap->message_count('Public folders/LearnAsHam');
             print $nonspamcount, " Notspam to process\n" if $debug;
    }

    # Process the spam mailbox
    $imap->select('Public Folders/LearnAsSpam');
    my @msgs = $imap->search("ALL");
    for (my $i=0;$i <= $#msgs; $i++)
    {
             # export message to file
             $imap->message_to_file($spamwebclient.".zarafa.".time.".".$i,$msgs[$i]);

             # delete processed message
             $imap->delete_message($msgs[$i]);
             open("placeholder",'>'.$spamwebclient) or die "Can't create placeholder $!";
             close("placeholder");
    }

    $imap->expunge();
    $imap->close();

    # Process the not-spam mailbox
    $imap->select('Public Folders/LearnAsHam');
    my @msgs = $imap->search("ALL");
    for (my $i=0;$i <= $#msgs; $i++)
    {
             $imap->message_to_file($notspamwebclient.".zarafa.".time.".".$i,$msgs[$i]);

             # delete processed message
             $imap->delete_message($msgs[$i]);
             open("placeholder",'>'.$notspamwebclient) or die "Can't create placeholder $!";
             close("placeholder");
    }
    $imap->expunge();
    $imap->close();

    $imap->logout();


    Make sure it's executable
    Code:

    chmod a+x /usr/bin/Zarafa-sa-learn.pl



  8. Create cron script to regularly pull mail from the Public folder into the Spam training system
    Create /etc/cron.d/zarafa-spamassassin
    Code:

    # Extract mail from spam folders for training every 10minutes from 8:00 to 22:00 during weekdays
    */10 8-22 * * 1-5 root /usr/bin/Zarafa-sa-learn.pl



  9. Test! you should now be able to login via webmail or Outlook, and drag (move) mail that isnt' being correctly recognised as SPAM into the LearnAsSpam folder and ClearOS will 'auto-learn' it. The same applies for HAM (mail incorreclty marked as SPAM). Note that mail is deleted after it is moved into the Public Folders so keep a copy if you need it.


Enjoy!
Tim Burgess
Moderator
Posts: 6601
graph
User Offline Click here to see the profile of this user
Last Edit: 2011/01/17 05:28 By timb80.
The administrator has disabled public write access.
 
#23322
Re:Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 7 Months ago  
Thanks Tim. I'll try this week.

Congratulations
weimar
Expert Boarder
Posts: 99
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#24772
Re:Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 6 Months ago  
Giving it a try
I'll try to report back with my results.
Thomas Thillerup
Fresh Boarder
Posts: 4
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#24827
Re:Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 6 Months ago  
It seemed like this one chrashed my box... I installed it as per guide, tossed a few hundred mails in the LearnAsSpam, sat back and waited... time went by, nothing happend, then it hit's me... it's past 22, oh well, turned in and the next morning i went of to work. And at 9 I wanted to check up on the folder status, was it empty og was it still full..? Guess what.. I could not get in contact with my server..?? Dang!!! Went home with the thoughts of a long evening of error correcting in mind. My server is an old notebook that serves my needs, and it's alway turned on (Hey it has a server function so it has to be) and as a result it gets a bit hot. Nothing alarming, just about 37 degrees (C). The day before installing from this guide, I had som other work to do on the notebook. I forgot to close the screen down after i was done. Did I mention I have a cat...? He's a one year old helfbreed Maincoon, and he just looooves those hot places.... He lay down on the keyboard, and by doing that also pressed down on the powerbutton... And with a purr from the cat and a hum from the notebook, my server went offline...

Got the cat off the notebook and the notebook powered on again, and presto... the cron job empties the folder as expected, and it seems like the Bayesian learning is taking place.
Thomas Thillerup
Fresh Boarder
Posts: 4
graphgraph
User Offline Click here to see the profile of this user
Last Edit: 2011/02/24 14:37 By thill.
The administrator has disabled public write access.
 
#24830
Re:Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 6 Months ago  
LOL glad it's working, thanks for the feedback
Tim Burgess
Moderator
Posts: 6601
graph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#27053
Re: Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 4 Months ago  
Tim,

I'm having trouble connecting to LDAP using LdapAdmin. It always returns "LDAP Server Down". I know its up and running. Can you explain a little more on exactly what I need to enter in it.

Thanks
Kevin Farmer
Junior Boarder
Posts: 32
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#27054
Re: Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 4 Months ago  
Hi Kevin, is your LDAP publish policy enabled? if not, then you'll only be able to connect via the local loopback interface (127.0.0.1).. You can change this setting from the LDAP menu in the webconfig
Tim Burgess
Moderator
Posts: 6601
graph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#27057
Re: Zarafa Webmail - Spam Training / Bayesian Learning 3 Years, 4 Months ago  
Got it... I have some FAT Fingers... Great work.. I'm loving Zarafa.
Kevin Farmer
Junior Boarder
Posts: 32
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#32041
Re: Zarafa Webmail - Spam Training / Bayesian Learning 2 Years, 12 Months ago  
Thanks for the great work, Tim - on Zarafa especially and this training feature in particular. I'm up and running Zarafa 7.01. Very much appreciate your contribution of so much time and talent.
DaveBurkholder
Expert Boarder
Posts: 157
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
#32376
Re:Zarafa Webmail - Spam Training / Bayesian Learning 2 Years, 11 Months ago  
hi,Tim!

[zarafa 7.01] some wrong when restart ladp before add 'zarafa.schem' to sldap.conf:

[root@mail openldap]# service ldap start

Checking configuration files for slapd: [FAILED]
/etc/openldap/schema/zarafa.schema: line 182 objectclass: AttributeType not found: "mail"
slaptest: bad configuration file!

my sldap.conf:
-----------------------------------------------------------------------------
access to dn.regex="(.*,)?cn=internal,dc=mail,dc=post88,dc=net"
by group/kolabGroupOfNames="cn=admin,cn=internal,dc=mail,dc=post88,dc=net" write
by group/kolabGroupOfNames="cn=maintainer,cn=internal,dc=mail,dc=post88,dc=net" write
by self write
by dn="cn=nobody,cn=internal,dc=mail,dc=post88,dc=net" read #The line 182
by anonymous auth stop
-----------------------------------------------------------------------------
hope your help,thx!
jplele
Fresh Boarder
Posts: 5
graphgraph
User Offline Click here to see the profile of this user
The administrator has disabled public write access.
 
Go to topPage: 1234
  get the latest posts directly to your desktop