{"id":27,"date":"2009-10-01T15:12:55","date_gmt":"2009-10-01T12:12:55","guid":{"rendered":"http:\/\/www.howto.gr\/wp\/?p=27"},"modified":"2009-10-01T15:45:49","modified_gmt":"2009-10-01T12:45:49","slug":"bacula-network-backup","status":"publish","type":"page","link":"http:\/\/www.howto.gr\/wp\/bacula-network-backup\/","title":{"rendered":"Bacula Network Backup"},"content":{"rendered":"

What is the most important and boring job an administrator has to do? Backup! You may have the super duper high availability cluster, redundant storage arrays with multiple raid levels. But what if you make a mistake? Do you believe mistakes are rare? Count how many typos you have during a typical day working on a Unix\/Linux console… I suppose it is not that difficult to type \u201crm -fr . \/\u201d instead of \u201crm -fr .\/\u201d …. So we need to have regular, reliable, manageable and carefully planned backup strategy.<\/p>\n

I believe it is common for lots of administrator around the world (including me!) a situation where you have:<\/p>\n

    \n
  1. Lots of servers (at least 3)<\/li>\n
  2. Servers in remote sites<\/li>\n
  3. Low or no budget at all (for buying very very very expensive professional backup software)<\/li>\n
  4. Various OS platforms.<\/li>\n<\/ol>\n

    The best Open Source solution out there is Bacula (http:\/\/www.bacula.org<\/a>). Bacula is an open source set of applications that provide functionallity equivalent of the one of expensive commercial products. Unfortunately it does not have a GUI for configuring it and it looks really frightening when you first touch it. However it is more a matter of getting used to its logic rather than difficulty in configuration (in the contraty, configuration itself is really straight forward!). Below I will present you a real case scenario (already working in production) and a bacula success story.<\/p>\n

    Get in touch with Bacula terminology<\/h1>\n

    Bacula is based on the following 4 terms. Director, File Daemon, Storage Daemon, Console. Each one is independent and can exist on one machine either by itself either in addition with others. Console is pretty obvious. It is the \u201chost\u201d that can connect to bacula director and issue commands. File Daemon is the responsible daemon that actually performs the backup. It reads data from the disk and send them to the Storage daemon. Storage Daemon is responsible of writing data to the backup device. It receives data from a File Daemon and writes them to the specified backup device or file. Finally, Director is the responsible daemon to manage everything on bacula. It handles media pools, backup jobs, backup schedules and also controls File and Storage Daemons.<\/p>\n

    It is beyond the scope of this How To, to provide further explanation to what is Director, File and Storage Daemons, as well as Media Pools, Volumes and various other terms generally used when talking about backups. Bacula Manual is a perfect manual with lots of info to make it clear to you!<\/p>\n

    Network Topology<\/h1>\n

    We have a main site and two remote sites. Our strategy is the following. On the first Sunday of each month we have a full system backup. On the 2nd to 4th Sunday of each month we have a differential backup and every day we have an incremental system backup. Moreover, ouranos, lahesis, okeanos and callisto are being backed up on callisto. Atropos is being backed up on atropos (on an external disk) and on sivyla. Sivyla is being backed up on sivyla (on an external disk) and on atropos. The reason for the remote sites to be backed up besides one to the other, but also locally is just speed in case of a small restore or the survival of the external disk in case of system failure. Everything is managed from callisto. So we have the following schema:<\/p>\n

    \"Bacula<\/a>
    Bacula Network Diagram<\/figcaption><\/figure>\n

    Overall the have following servers and roles<\/p>\n\n\n\n\n\n\n\n\n
    Callisto<\/td>\nDirector (dir), File Daemon (fd), Storage Daemon (sd), Console<\/td>\n<\/tr>\n
    Ouranos<\/td>\nFd<\/td>\n<\/tr>\n
    Lahesis<\/td>\nFd<\/td>\n<\/tr>\n
    Okeanos<\/td>\nFd<\/td>\n<\/tr>\n
    Atropos<\/td>\nSd, Fd<\/td>\n<\/tr>\n
    Sivyla<\/td>\nSd, Fd<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    <\/h1>\n

    Configuration Logic<\/h1>\n

    The heart of Bacula is the Director. As a consequence it has the most difficult and complicated configuration. Before you start make sure you have a piece of paper and a pencil handy to keep track of you configuration choices. I will not go deep in details about every configuration option but rather I would like to make you familiar with the logic behind the configuration file. We will see a complete configuration regarding Callisto. First of all you must have a Media Pool to write your data. We choose the name \u201cCallistoPool\u201d to refer to this pool. We also want to autolabel the media and use a name convention like CallistoVolXXX (where XXX is a 3-digit number starting from 000). Finally we want a volume lifetime of 1 year. Having these in mind we have the following snippet of configuration:<\/p>\n

    Pool {\r\n  Name = CallistoPool\r\n  LabelFormat = \"CallistoVol\"\r\n  Pool Type = Backup\r\n  Recycle = yes # Bacula can automatically recycle Volumes\r\n  AutoPrune = yes # Prune expired volumes\r\n  Volume Retention = 365 days # one year\r\n  Accept Any Volume = yes # write on any volume in the pool\r\n}<\/pre>\n
    As we have our media pool it\u2019s time to define our storage device. In general we define a way to talk to a specific storage daemon. The IP address to connect and the port. For security reasons we must also have the password to connect to the storage daemon. Finally we choose in which device we want the storage daemon to write (we must have already configure this to the storage daemon configuration) and which media pool to use. So the configuration snippet is:<\/p>\n
    Storage {\r\n  Name = FileCallisto\r\n  Address = callistoprv.howto.gr # N.B. Use a fully qualified name here\r\n  SDPort = 9103\r\n  Password = \"errgd452345243srqwrdwfcsvmoxvzidfhvpsarhjps\"\r\n  Device = FileStorageCallisto\r\n  Media Type = CallistoVol\r\n}<\/pre>\n
    Now it\u2019s time to configure the client to back up! First of all we must define a name to refer to this client, its ip address, the tcp port to connect to and the password. Other options depend heavily on our backup policy.<\/p>\n
    Client {\r\n  Name = callisto-fd\r\n  Address = callistoprv.howto.gr\r\n  FDPort = 9102\r\n  Catalog = MyCatalog\r\n  Password = \"o9h3Asdwadasdq123rsdfV9lzIwnsdasdasDkUpz120bIM\" # password for FileDaemon\r\n  File Retention = 30 days # 30 days\r\n  Job Retention = 6 months # six months\r\n  AutoPrune = yes # Prune expired Jobs\/Files\r\n}<\/pre>\n
    It should be obvious that our next step is to define what we want to backup on the previously specified client. So we configure filesets. Suppose that callisto has only three mounted file systems. Root (\/), boot (\/boot) and var (\/var). We name this fileset \u201cCallisto_Full_fs\u201d and enable the option to sign the files using MD5. Of course it is useless to backup \/proc, \/tmp and probably various other directories!<\/p>\n
    FileSet {<\/pre>\n
      Name = \"Callisto_Full_fs\"<\/pre>\n
      Include {<\/pre>\n
        Options {<\/pre>\n
          signature = MD5<\/pre>\n
        }<\/pre>\n
        File = \/<\/pre>\n
        File = \/boot<\/pre>\n
        File = \/var<\/pre>\n
      }<\/pre>\n
      Exclude {<\/pre>\n
        File = \/proc<\/pre>\n
        File = \/tmp<\/pre>\n
        File = \/.journal<\/pre>\n
        File = \/.fsck<\/pre>\n
      }<\/pre>\n
    }<\/pre>\n
    Up until now we know WHERE and WHAT to backup. Let\u2019s define WHEN! We create a schedule with the name \u201cWeeklyCycle\u201d and we define to have from Monday to Saturday an incremental backup. On the first Sunday of each month we have a full backup and from the second to the fifth Sunday we want to have a differential backup. All these should be done at 1 am every night.<\/p>\n
    Schedule {<\/pre>\n
      Name = \"WeeklyCycle\"<\/pre>\n
      Run = Full 1st sun at 1:05<\/pre>\n
      Run = Differential 2nd-5th sun at 1:05<\/pre>\n
      Run = Incremental mon-sat at 1:05<\/pre>\n
    }<\/pre>\n
    So our last step is to take all the information above and create a job. Bacula runs jobs, so when we know where, what and when we are ready to create a job for bacula. We name our job \u201cCallisto_Full\u201d. It should connect to the client \u201ccallisto-fd\u201d, backup all the files defined in the fileset \u201cCallisto_Full_fs\u201d and the backup should be written to the storage daemon \u201cFileCallisto\u201d. Next we define some extra configuration options mostly to define the behavior of our job.<\/p>\n
    Job {<\/pre>\n
      Name = \"Callisto_Full\"<\/pre>\n
      Type = Backup<\/pre>\n
      Client = callisto-fd<\/pre>\n
      FileSet = \"Callisto_Full_fs\"<\/pre>\n
      Storage = FileCallisto<\/pre>\n
      Pool = CallistoPool<\/pre>\n
      Messages = Standard<\/pre>\n
      Where = \/tmp\/bacula-restores<\/pre>\n
      JobDefs = \"FullBackupJob\"<\/pre>\n
      Write Bootstrap = \"\/var\/bacula\/callisto.bsr\"<\/pre>\n
    }<\/pre>\n
    Congratulations! You have created your first backup job. Of course there is still more things to do like configuring the Storage Daemon and the File Daemon but it must be really straight forward now. Do your homework and of course have Bacula Manual next to you, it is extremely useful!!! Below I provide you with full configuration for the network and the schema mentioned above.<\/p>\n
    Good Luck!<\/p>\n
    bacula-dir.conf<\/h3>\n
    ### Director ###<\/pre>\n
    Director<\/pre>\n
    { # define myself<\/pre>\n
    Name = callisto-dir<\/pre>\n
    DIRport = 9101 # where we listen for UA connections<\/pre>\n
    QueryFile = \"\/etc\/bacula\/query.sql\"<\/pre>\n
    WorkingDirectory = \"\/var\/bacula\/working\"<\/pre>\n
    PidDirectory = \"\/var\/run\"<\/pre>\n
    Maximum Concurrent Jobs = 1<\/pre>\n
    Password = \"trUtd567tqxHxG9jkldDElX+yr13fx6VBposdflLHl4\" # Console password<\/pre>\n
    Messages = Daemon<\/pre>\n
    }<\/pre>\n
    #########################################<\/pre>\n
    ########### Job Definitions #############<\/pre>\n
    #########################################<\/pre>\n
    JobDefs {<\/pre>\n
    Name = \"FullBackupJob\"<\/pre>\n
    Type = Backup<\/pre>\n
    Level = Incremental<\/pre>\n
    Schedule = \"WeeklyCycle\"<\/pre>\n
    # Storage = File<\/pre>\n
    Messages = Standard<\/pre>\n
    # Pool = Default<\/pre>\n
    Priority = 10<\/pre>\n
    }<\/pre>\n
    JobDefs {<\/pre>\n
    Name = \"WinBackup\"<\/pre>\n
    Type = Backup<\/pre>\n
    Level = Full<\/pre>\n
    Schedule = \"WeeklyCycle\"<\/pre>\n
    Messages = Standard<\/pre>\n
    Priority = 10<\/pre>\n
    }<\/pre>\n
    #########################################<\/pre>\n
    ################# Jobs ##################<\/pre>\n
    #########################################<\/pre>\n
    #--- Linux Forthnet<\/pre>\n
    Job {<\/pre>\n
    Name = \"Callisto_Full\"<\/pre>\n
    Type = Backup<\/pre>\n
    Client = callisto-fd<\/pre>\n
    FileSet = \"Callisto_Full_fs\"<\/pre>\n
    Storage = FileCallisto<\/pre>\n
    Pool = CallistoPool<\/pre>\n
    Messages = Standard<\/pre>\n
    Where = \/tmp\/bacula-restores<\/pre>\n
    JobDefs = \"FullBackupJob\"<\/pre>\n
    Write Bootstrap = \"\/var\/bacula\/callisto.bsr\"<\/pre>\n
    }<\/pre>\n
    Job {<\/pre>\n
    Name = \"Ouranos_Full\"<\/pre>\n
    Type = Backup<\/pre>\n
    Client = ouranos-fd<\/pre>\n
    FileSet = \"Ouranos_Full_fs\"<\/pre>\n
    Storage = FileOuranos<\/pre>\n
    Pool = OuranosPool<\/pre>\n
    Messages = Standard<\/pre>\n
    Where = \/tmp\/bacula-restores<\/pre>\n
    JobDefs = \"FullBackupJob\"<\/pre>\n
    Write Bootstrap = \"\/var\/bacula\/ouranos.bsr\"<\/pre>\n
    }<\/pre>\n
    Job {<\/pre>\n
    Name = \"Lahesis_Full\"<\/pre>\n
    Type = Backup<\/pre>\n
    Client = lahesis-fd<\/pre>\n
    FileSet = \"Lahesis_Full_fs\"<\/pre>\n
    Storage = FileLahesis<\/pre>\n
    Pool = LahesisPool<\/pre>\n
    Messages = Standard<\/pre>\n
    Where = \/tmp\/bacula-restores<\/pre>\n
    JobDefs = \"FullBackupJob\"<\/pre>\n
    Write Bootstrap = \"\/var\/bacula\/lahesis.bsr\"<\/pre>\n
    }<\/pre>\n
    #--- Linux Foreign<\/pre>\n
    Job {<\/pre>\n
    Name = \"Atropos_Full\"<\/pre>\n
    Type = Backup<\/pre>\n
    Client = atropos-fd<\/pre>\n
    FileSet = \"Atropos_Full_fs\"<\/pre>\n
    Storage = FileAtropos<\/pre>\n
    Pool = AtroposPool<\/pre>\n
    Messages = Standard<\/pre>\n
    Where = \/tmp\/bacula-restores<\/pre>\n
    JobDefs = \"FullBackupJob\"<\/pre>\n
    Write Bootstrap = \"\/var\/bacula\/atropos.bsr\"<\/pre>\n
    }<\/pre>\n
    Job {<\/pre>\n
    Name = \"Atropos_To_Sivyla_Full\"<\/pre>\n
    Type = Backup<\/pre>\n
    Client = atropos-fd<\/pre>\n
    FileSet = \"Atropos_Full_fs\"<\/pre>\n
    Storage = FileAtropos_Sivyla<\/pre>\n
    Pool = AtroposPool_Sivyla<\/pre>\n
    Messages = Standard<\/pre>\n
    Where = \/tmp\/bacula-restores<\/pre>\n
    JobDefs = \"FullBackupJob\"<\/pre>\n
    Write Bootstrap = \"\/var\/bacula\/atropos_to_sivyla.bsr\"<\/pre>\n
    }<\/pre>\n
    Job {<\/pre>\n
    Name = \"Sivyla_Full\"<\/pre>\n
    Type = Backup<\/pre>\n
    Client = sivyla-fd<\/pre>\n","protected":false},"excerpt":{"rendered":"
    What is the most important and boring job an administrator has to do? Backup! You may have the super duper high availability cluster, redundant storage arrays with multiple raid levels. But what if you make a mistake? Do you believe mistakes are rare? Count how many typos you have during a typical day working on […]<\/p>\nRead More \u2192<\/a>","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"_links":{"self":[{"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/pages\/27"}],"collection":[{"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":14,"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/pages\/27\/revisions"}],"predecessor-version":[{"id":33,"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/pages\/27\/revisions\/33"}],"wp:attachment":[{"href":"http:\/\/www.howto.gr\/wp\/wp-json\/wp\/v2\/media?parent=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}