My Personal Computing Network

For my research and teaching purposes, I conducted design of materials regularly. I also often constructed stochastic models to simulate experimental data acquired in my lab. To meet those various demands, I concluded that a personal computing facility will be useful to me. So in past several months, I began to survey and build a personal computing network with open source softwares. However, because of my poor memory, I am unable to remember what are the scripts I used to build the network and the simulation packages after several weeks working on other business. To avoid such inconveniences, I decide to create these webpages to help me maintain the computing facility in hope of simply copying and pasting relevant scripts from the webpages.

In this web page, I first describe the architecture of my computing network and the building procedures of the computing network. In the next webpages, I will describe how I install some software libraries and packages, which are useful for my computing purpose. I hope these webpages will be helpful to you too by saving your time.

I. The Architecture of My Computing Network


My personal computating network consists of several x86_64-linux machines (with ubuntu16.04 LTS) scattered in different rooms. I hope to connect these spared computers to form a cluster for my distributed computing applications. For simplicity, I will just use three computers to illustrate the network architecture:
Machine 1 (hostname hp-xw6000): master DNS/web server with eth0 (ip=140.113.78.123)
Machine 2 (multicore cpu+nvidia gpu with hostname dell-t7500): slave DNS server with eth1 (ip=140.113.78.124) and a computing server with eth0 (an internal ip=10.42.0.1)
Machine 3 (multicore cpu+nvidia gpu with hostname dell-m4800): computing client node with eth0 (an internal ip=10.42.0.2) and a wireless router wlan0 for internet access.

$ : Ubuntu shell response

(A) Install BIND master DNS on machine 1 (hp-xw6000)

1) $ sudo nano /etc/hosts --> Edit this file to look like this file pdf.

2) Insert hp-xw6000 into the hostname file using the command:
 $ sudo nano /etc/hostname

3) Install Bind on on the Server:
  $ sudo apt-get update
  $ sudo apt-get install bind9

4) $ sudo nano /etc/bind/named.conf.options --> Configure the master DNS Bind server to be similar to pdf.

5) $ sudo cp /etc/bind/db.local /etc/bind/db.jyhuang.idv.tw --> Create a forward zone file /etc/bind/db.jyhuang.idv.tw.

6) $ sudo nano /etc/bind/db.jyhuang.idv.tw --> Edit the forward zone file to be similar to pdf and then save the file.

7) $ sudo cp /etc/bind/db.127 /etc/bind/db.140.113.78--> Create a reverse zone file.

8) $ sudo nano /etc/bind/db.140.113.78 --> Edit the reverse zone file to be similar to pdf and then save the file.

9) Testing the Files with
  $ sudo named-checkconf
  $ sudo named-checkzone jyhuang.idv.tw /etc/bind/db.jyhuang.idv.tw
  $ sudo named-checkzone 78.113.140.in-addr.arpa /etc/bind/db.140.113.78
  If no errors occurs, then restart the bind9 server with
  $ sudo service bind9 restart

(B) Install BIND slave DNS on machine 2 (dell-t7500)

1) $ sudo nano /etc/hosts --> Edit the file to be similar to pdf.

2) Insert dell-t7500 into the hostname file
  $ sudo nano /etc/hostname

3) To prevent unnecessary interference, I remove the network-manager on machine 2 and allow the network interfaces and resolv.conf to function. Prepare the networking interfaces file with the command:
  $ sudo nano /etc/network/interfaces --> Edit the file to be similar to pdf.

4) $ sudo nano /etc/resolv.conf --> Check out the file to have
  # Generated by NetworkManager
  nameserver 127.0.0.1
  nameserver 8.8.8.8
  nameserver 140.113.1.1

5) Install Bind9 on on the Server:
  $ sudo apt-get update
  $ sudo apt-get install bind9

6) $ sudo nano /etc/bind/named.conf.options --> Configure the slave DNS Bind server to be similar to pdf.

7) $ sudo nano /etc/bind/named.conf.local --> Configuring the local configuration file to be similar to pdf.

8) The forward and reverse zone files of jyhuang.idv.tw will be directly transferred from machine 1. We only have to prepare the forward zone file of ompi.net as pdf,
  and the reverse zone file of ompi.net as pdf.

9) Testing the Files with
  $ sudo named-checkconf
  If no errors occurs, then restart the bind9 server with
  $ sudo service bind9 restart

(C) Machine 3 (dell-m4800)

1) The networking of machine 3 possesses a wireless wlan0 (3c:a9:f4:b2:3e:70) and a wired eth0 (34:e6:d7:1d:3e:ca) network connections. We will use network manager to only manage wlan0. Prepare /etc/NetworkManager/NetworkManager.conf as pdf.

2) sudo nano /etc/network/interfaces --> Edit the file to be similar to pdf.

3) $ sudo nano /etc/hosts --> Check the host file to have
  127.0.0.1 localhost
  10.42.0.1 dell-t7500.ompi.net dell-t7500
  10.42.0.10 dell-m4800.ompi.net dell-m4800

To Top

II. Install Computing Network


A Beowulf Cluster is a collection of networked computers which can be used for parallel computations. They consist of commodity hardware running open source softwares coupled with MPI (Message Passing Interface). A standard set up will consist of one master node which will control a number of slave nodes. The slave nodes are typically headless and generally all access the same files from a server.

Setting up the Ubuntu cluster machines
(A) On the computation slave node (dell-t7500) :
  (1) $ apt-get install openssh-server
  (2) $ cd /etc/ssh
    $ sudo nano sshd_config
    Check the following lines to have the correct setting:
    RSAAuthentication   yes
    PubKeyAuthentication   yes
    AuthorizedKeysFile   %h/.ssh/authorized_keys
    PasswordAuthentication   no
  (3) $ sudo /usr/sbin/sshd

(B) On the computation master node (dell-m4800) :
  (1) $ apt-get install openssh-client

  (2) While logged in as an open mpiuser (e.g., xxyyzz), create SSH public/private key pairs, which are to be stored in the folder /home/xxyyzz/.ssh/ -->
     $ mkdir .ssh & cd .ssh
     $ ssh-keygen -t rsa

  (3) Keep the private key id_rsa intack and then copy the public key to the authorized_keys in the subdirectory on each slave node. -->
     $ cp id_rsa.pub   authorized_keys
     $ scp /home/xxyyzz/.ssh/id_rsa.pub   xxyyzz@dell-t7500: .ssh/authorized_keys

  (4) Correct file permissions (This will also need to be done on the slave nodes if not sharing the home folder)
     $ chmod 700 /home/xxyyzz/.ssh
     $ chmod 600 /home/xxyyzz/.ssh/authorized_keys

  (5) Use ssh-agent to remember the password while logged in as mpiuser xxyyzz
     $ eval `ssh-agent`
     Tell ssh-agent the password for the SSH key
     $ ssh-add ~/.ssh/id_rsa

  (6) Test by logging into one of the slave nodes, while logged in as mpiuser xxyyzz
     $ ssh xxyyzz@dell-t7500

This shouldn't ask for a password. If it is not working, check the known_hosts file on the master note. Keep only the newest line of host.

To simplify the task, I had installed third-party softwares for ubuntu linux and open source libraries to ensure similar environment on each computing node. These software resource will be needed to support some useful open source simulation packages for my applications.

To Top