########################################################################### # Linux Extended Notes # # Anuradha Weeraman, 08 November 2003 # # Adapted from "Linux Server Hacks", Rob Flickenger, O'Reily & Assoc. # # $Id: linux-extended-notes.txt,v 1.1 2004/06/02 21:17:53 anuradha Exp $ # ########################################################################### --- Secure CVS export CVS_RSH=`which ssh` export CVSROOT=:ext:anuradha@host:/var/lib/cvs cvs checkout project ---------- --- Locking and Unlocking a terminal skill -STOP tty2 skill -CONT tty2 ---------- --- RCS mkdir RCS # create file1 ci -i file1 # Checks file into repos and removes it from dir co -l file1 # Check out file with a lock # make changes ci -u file1 # Check in and unlock rlog file1 rcs2log file1 # make more changes rcsdiff file1 rcsdiff -r1.1 file1 # Create diff between current ver and 1.1 rcsdiff -r1.1 -r1.2 file2 # Create diff between 1.1 and 1.2 ---------- ----- Parallel builds On SMP systems, build can be performed in parallel make -j2 bzImage make -j4 bzImage make -j9 bzImage 2 is a good tradeoff, higher numbers tend to carry a bit more overhead. ---------- --- CDPATH export CDPATH=.:/usr/local This will search /usr/local for directories to 'cd' into. Very similar to the PATH, MANPATH environment variable. ---------- --- sudo /etc/sudoers syntax: user machine=(effective user) command anuradha ALL=(ALL) ALL ---------- --- procps snice +5 luser skill -KILL bash pkill -KILL -uanuradha -ttty2 bash pgrep httpd vmstat ---------- --- ulimit ulimit -a ulimit -f 100 # Maximum file size = 100k ulimit -f unlimited ulimit -u 10 # Maximum number of processes ---------- --- passwd passwd -l # Locks user passwd -u # Unlocks user passwd -d # Removes passwd, user will be able to login without one passed -e # Expires passwd, user forced to change it when logging in ---------- --- High memory Above 960MB, the kernel must be compiled with high memory support and the following added to the kernel boot parameters in the appropriate bootloader. Assuming the system has two GB of memory : Lilo: append="mem=2048M" Grub: kernel /boot/vmlinuz-2.4.20 mem=2048M Loadlin (?): c:\loadlin c:\kernel\vmlinuz root=/dev/hda3 ro mem=2048M Linux supports upto 64GB of RAM using the PAE (Physical Address Extension) in new Intel processors (Pentium Pro and above). This option might crash machines that do not have PAE and is left off by default. 4GB does not need PAE. ---------- --- hdparm Benchmark: hdparm -Tt /dev/hda Supported options: hdparm /dev/hda hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda -X66 : UltraDMA 2 (Starts from 64) -d1 : DMA mode -u1 : Un mask irq -m16 : Multiple sector count -c3 : I/O support ---------- --- CVS tags, branches and watching cvs tag tagname filename cvs tag tagname cvs checkout -r tagname cvs update -r tagname # Branches cvs tag -b branch cvs checkout -r branch cvs update -r branch cvs checkout module cvs update -j branch # CVS does not have support for locking files cvs watch on (files) cvs watch off (files) cvs watch add (files) cvs watch remove (files) cvs watch add -a edit|unedit|commit|all (files) cvs watch remove -a edit|unedit|commit|all (files) ---------- --- CVS pserver 1) Add a user, anoncvs. 2) Lock the user, passwd -l anoncvs 3) Set the user's shell to /bin/true 4) Set the homedir to something bogus, like /var/empty 5) Add the following to CVSROOT/passwd anonymous:23MLN3ne5kvBM:cvsanon # The password is "anonymous" 6) To make sure that the anoymous user cannot make changes, add the following to CVSROOT/readers anonymous 7) To prevent regular users from logging into the pserver, add the following to CVSROOT/config SystemAuth=no This will have no effect on logins via ext and ssh, and only the CVSROOT permissions will provide access control. 8) The system can then be configured to accept pserver connections by adding the following to /etc/inetd.conf. pserver stream tcp nowait root /usr/bin/cvs cvs \ --allow-root=/usr/local/cvsroot pserver 9) skill -HUP inetd # re-read inetd.conf 10) Test the pserver by setting CVSROOT to :pserver:anonymous@your.machine:/usr/local/cvsroot 11) Before making check outs you first need to login to cvs with cvs login which will store the login credentials in ~/.cvspass. When you are don you can logout by cvs logout ---------- --- tar & ssh If you want to make a copy of a directory structure on a remote machine, it can be easily done by : tar zcvf - /home/anuradha/ | ssh remote "cat > anuradha.tar.gz" tar zcvf - anuradha/ | ssh remote \ "cd /home; mv anuradha anuradha.backup; tar zpxvf -" Untar to a remote machine : ssh remote "cd /target/dir; tar zpxvf -" < local.tar.gz Vice versa ssh remote "cat remote.tar.gz" | tar zpxvf - If you have trouble trying these, make sure that nothing is being written by ~/.bashrc on the remote machine. Its better to move "fortune" stuff into ~/.bash_profile or ~/.bash_login as its only needed if there's a user login. ---------- --- rsync rsync -ave ssh remote:/home/ftp/pub/ /home/ftp/pub/ Rsync will only keep the missing files in sync with the local copy. To keep completely in sync, that is delete files that have been removed in the source machine, include the --delete option. rsync -ave ssh --delete remote:/home/ftp/pub/ /home/ftp/pub/ rsync can be used to replicate web content for instance for load balancing. rsync -ae ssh master:/var/www/ /var/www/ A cron job can be added to perform this every so often, like 5 minutes. But if an rsync takes longer to perform, it can enter into a downward spiral when an rsync takes longer than the allotted interval increasing the load and progressively bringing the system to a halt. ---------- --- pax PAX stands for Portable Archive Exchange and serves as a pact between the tar and cpio enthusiasts. pax -wvf dir.pax dir # -w = write To see the contents: pax -f dir.pax | more pax -vf dir.pax | more # ls -l type listing To restore : cd destination pax -rvf /source/dir.pax You can use pax to restore both tar and cpio archives. Pax will automatically detect them for you. To deflate dir.tar.gz: pax -rzvf dir.tar.gz You can use pax interactively using the -i switch. pax -rif backup.tar.gz pax can also be used to recursively copy directories. cd source mkdir target pax -rw . target To do this interactively : pax -rwi . target You should not do : cd source mkdir target pax -rw . target As that would put pax into an infinite loop. Example backup policy: 1) Full backup on Monday cd /home/anuradha pax -wzf /var/backups/Monday.pax . 2) Incremental backups, the rest of the week cd /home/anuradha pax -wzf /var/backups/Tuesday.pax . -T 0000 -T indicates that files modified since midnight should be backed up. To restore all files except 'file3' : pax -rvf backup.pax -c 'file3' To restore a particular file : par -rvf backup.pax -n 'file1' Wildcards can be used for these. ---------- --- Bootsector dd if=/dev/[hs]da of=bootsector.bin bs=512 count=1 dd if=bootsector.bin of=/dev/[hs]da ---------- --- Bash Check error codes : mount /floppy if (( $? )); then echo "Failed!" else echo "Successful!" fi Check root : if (( `id -u` != 0 )); then echo "Not root" fi ---------- --- ISO mkisofs -r /home/anuradha > anuradha.iso -r = rock ridge extensions, preserves long file names and permissions To make an ISO of an existing data cd : dd if=/dev/cdrom of=image.iso You can make this go faster by manipulating the bs parameter : dd if=/dev/cdrom of=image.iso bs=10k To mount an iso on the loopback device : mount -o loop,ro -t iso9660 ./image.iso /mnt You might need to say 'modprobe loop' if it isn't compiled into the kernel already. To burn an ISO to CD-R : cdrecord -v speed=12 dev=0,0,0 -data image.iso CD-RW : cdrecord -v speed=12 dev=0,0,0 blank=fast -data.image.iso You can sometimes burn straight from cdrom to cdwriter, provided you have both the drives : dd if=/dev/hdb | cdrecord -v speed=12 dev=0,0,0 fs=8m -data - The fs option makes the write FIFO bigger to offset any momentary hiccups. Also : mkisofs -r /home/anuradha | cdrecord -v speed=12 dev=0,0,0 fs=8m -data - Real time network burn : mkisofs -r /home/anuradha | ssh remote "cdrecord -v \ speed=12 dev=0,0,0 fs=8m -data -" Copying a local CD to a network burner in real time : dd if=/dev/cdrom | ssh remote "cdrecord -v speed=12 dev=0,0,0 fs=8m \ -data -" ---------- --- syslogd You would have seen "-- MARK --" lines in your system logs. Its syslog's way of "touching base" and is there to indicate that syslog is running even when it doesn't seem to be logging anything. This can be turned off by : killall syslogd; /usr/sbin/syslogd -m 0 Syslogd can be used to log for remote machines on the network. To do so : /usr/sbin/syslogd -m 0 -r This is a very susceptible service to random miscreants and must be protected. It can be easily done by the following iptables rule : iptables -A INPUT -t filter -p udp --dport 514 -s ! $LOCAL_NET -j DROP As this service listens on UDP port 514. ---------- --- watch When obsessively monitoring an output by rapidly pressing up-arrow key and enter, it would be easier to use the watch command. It even color codes changes in reverse and can be configured to display at regular intervals. watch 'vmstat' # redundant, as vmstat # can be used to do the same watch -d 'vmstat' # shows changes in reverse ---------- --- netstat To monitor open ports : netstat -l # ports that are listening netstat -lp # listening ports with the associated program Add the switch -n if you want numerical output only and the services not resolved. ---------- --- lsof Sometimes when you try to unmount a resource and it gives you "device is busy", how do you find out which processes are using it? lsof /mntpoint To find out all the files in use by a given process : lsof -p pid To do so by name, rather than pid : lsof -c apache By terminal : lsof /dev/tty2 Multiple switches are ORed by default. To AND them, use -a argument. lsof -u anuradha -c bash -a To see the open sockets and their associatd processes : lsof -i ---------- --- ngrep Network monitoring with ngrep : ngrep -q GET ngrep -qi rob@nocat.net port 25 ---------- --- Failover Servers set up to use round-robin dns load balancing can be easily configured for fail-over using virtual IP addresses. The nodes can monitor each other by occasionally pinging the neighbours and creating a virtual interface for the downed host if there is no reply. The new MAC address can be broadcast by using the send_arp utility from the High Availability Linux project. Once the machine is up, the virtual interface can be brought down. ---------- --- Network statistics ntop - For in depth statistical information on the network. Comes with integrated webserver on port 3000. httptop - Real time http statistics in the spirit of 'top'. httptop -f combined /usr/local/apache/logs/access_log httptop can be configured to monitor virtual hosts. Add the following to the httpd.conf : CustomLog /usr/local/apache/logs/combined-log vhost LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \%"{User-Agent}i\"" vhost and then : httptop -f vhost /usr/local/apache/logs/combined-log ---------- --- ssh, password-less logins To use ssh public-keys, you need to generate public/secret key pair. To do so: ssy-keygen -t rsa If you want to be able to login to remote machines without the use of passwords, you shouldn't enter a password when prompted for. Note that this would leave the generated private key open to anyone who is able to get their hands on it, but then that's the same with passwords. You might not want to do this in highly paranoid environments. The generation process will create the private/public key pairs : .ssh/id_rsa (private) .ssh/id_rsa.pub (public) To perform password-less logins : ssh server "mkdir .ssh; chmod 0700 .ssh" scp .ssh/id_rsa.pub server:.ssh/authorized_keys2 To speed up logins even further, the shell can be used to create aliases to machines and allow quick access : alias callisto='ssh callisto.domain.org' For the security conscious, there is an even more secure way of logging into remote machines without having to deposit one's unprotected private key in remote machines. For example, a scenario is given below : callisto$ ssh ganymede ganymede$ ssh io io$ This is possible by having the unprotected secret key in each of the machines. This would pose a problem for the paranoid. A better way would be to use the ssh-agent and configure it to Forward Requests. Add the following to ~/.ssh/config or /etc/ssh/ssh_config. ForwardAgent yes Start the ssh-agent and add the private keys : eval `ssh-agent` ssh-add When sshing from callisto to ganymede, ssh-agent will ask you for credentials and cache it. Later when you ssh from ganymede to io, ganymede will enquire callisto for valid user credentials relieving you of the burden of typing the password again. This is a more secure alternative. To use the ssh-agent in a GUI environment, it needs to be started before the windowing environment. ---------- --- X over ssh ssh supports X11 Forwarding natively. This can be accomplished in a number of ways. 1) ssh -X remotehost echo $DISPLAY > host:0.0 xeyes & 2) Add "X11Forwarding yes" to /etc/sshd_config. 3) Add "ForwardX11 yes" to ~/.ssh/config. ---------- --- ssh port forwarding ssh can be used to tunnel un-encrypted traffic and provide a high level of security. It accomplishes local forwarding by binding to a local port, performing encryption, sending the encrypted data to the remote end of the ssh connection, then decrypting it and sending it to the remote host and port you specify. An ssh tunnel is initiated with the -L switch (for Local). ssh -f -N -L110:mailhost:110 -l user mailhost -f : forks ssh into the background -N : tells ssh not to attempt to run a command on the remote host Optionally, the tunnel can be encrypted by the -C switch. You can specify as many -L switches as you need. This binds local port 110 via an ssh tunnel to the mailhost's POP port. This command would require root privileges as its attempting to bind to a privileged port. To connect to the mail server securely, all you have to do is configure the mail client to connect to the local host's port 110 rather than the mail host. All traffic will be transparently encrypted. To also forward outbound traffic : ssh -f -N -L110:mailhost:110 -L25:mailhost:25 -l user mailhost Obviously, once your mail leaves the mailhost it will be transmitted in the clear unless gpg or pgp is used. If you are already logged into a remote machine: anu@remote:~$ ~C ssh> -L8080:localhost:80 This will forward the local port 8080 to remote's port 80. You can also allow other remote users to connect to your forwarded port by using the -g switch. The -D switch allows ssh to be configured as a Socks 4 proxy ---------- --- framebuffer Load the appropriate framebuffer kernel module : modprobe vga16fb Consult /etc/fb.modes for an appropriate resolution that is supported by the framebuffer driver, and : fbset -fb /dev/fb0 640x480-75 ---------- --- Remote OS detection The ways of detecting what OS is running on remote machines. 1) The easiest way is to simply telnet to the host. playground~> telnet hpux.u-aizu.ac.jp Trying 163.143.103.12... Connected to hpux.u-aizu.ac.jp. Escape character is '^]'. HP-UX hpux B.10.01 A 9000/715 (ttyp2) login: 2) Telnet to ftp server, use syst command, download /bin/ls and see what architecture it has been built for. payfonez> telnet ftp.netscape.com 21 Trying 207.200.74.26... Connected to ftp.netscape.com. Escape character is '^]'. 220 ftp29 FTP server (UNIX(r) System V Release 4.0) ready. SYST 215 UNIX Type: L8 Version: SUNOS 3) Use the web server headers to determine architecture. playground> echo 'GET / HTTP/1.0\n' | nc hotbot.com 80 | egrep '^Server:' Server: Microsoft-IIS/4.0 playground> 4) DNS host info records can become quite handy. 5) Social engineering. 6) If the machine is listening on 161/udp (snmp) you can get lots of detailed info by using snmpwalk from the CMU snmp tools distribution. 7) Scanners like nmap use complex algorithms that can be successfully used to determine os including version numbers. the command would be : namp -sS -p 80 -O -v sucker.host.com ---------- --- NMAP There are a number of ways that nmap initiates connections with remote hosts to determine which ports are open : TCP Connect() scanning : easiest, very fast, non-privileged users can use it too, if the port is listening it will connect else it will fail, can be used in parallel and greatly reduce time for scanning, easily detectable and filtered, logs will show all opened and failed connections. TCP Syn scanning : also called half-open scanning, send SYN packet and wait for response, a SYN|ACK means port open and a RST means otherwise. once SYN|ACK is received an RST is sent to bring down the connection, hence half-open connection. fewer sites will log it, requires root privileges to pull it off. TCP FIN scanning : used when SYN isn't secure enough. some firewalls log SYN attempts but allow FIN packets. closed ports then send an RST on receiving a FIN packet while in open ports it just disappears into oblivion. M$ boxes are broken in this regard that they send RST packets regardless of the port state and can therefore be used to identify between NT and unix boxes. Fragmentation scanning : breaks up the probe packet into much smaller IP fragments that can make it harder for packet filters to determine what u are doing. some packet filters and firewalls which queue packets before filtering won't have this problem (like the CONFIG_IP_ALWAYS_DEFRAG option in Linux). TCP reverse ident scanning : using a bug/feature in the ident protocol its possible to find out the owner of a running daemon. this can only be used with a full TCP Connect() scan. FTP bounce attack : an interesting feature in the ftp protocol allowed users to proxy ftp connections and remain anonymous while doing a lot of damage. most ftp servers nowadays don't have this problem. UDP ICMP port unreachable scanning : UDP is used instead of TCP, protocol is simpler but scanning is significantly more difficult because open ports are required to send an acknowledgement and closed ports aren't required to send an error packet. most hosts send ICMP_PORT_UNREACH when UDP port is not open and by the process of exclusion determine which ports are open. UDP recvfrom() and write() scanning : while non-root users can't read port unreachable errors directly linux notifies the user indirectly when he tries to connect to a closed port the second time. this can be used know which ports are open and which are closed. ICMP echo scanning : since ICMP echo scanning does not have any port abstraction, this can only be used to see whether a particular host is up or not. ----------