Category Archives: Internals

The /etc/ directory

Leave a reply

In the last post we looked at /etc/passwd.  That file is in the /etc/ directory, which contains all sorts of configuration files.  This time, we’ll talk about that directory and some of the interesting files in it.

Like I said, it’s a configuration directory.  For example, your computer has an IP address.  If that’s a static IP address, then it’s going to be stored in a file in the /etc/ directory.

Every time you start your computer, it starts a lot of programs by default.  For example, it starts init, the mail program, the program that runs DNS, the printing system, cron, and more.  All of this is controlled by the files in /etc/.

There’s also some interesting files there.  For example, /etc/hostname.

Remember cron?  There’s a system file that runs all of the cron jobs for the system.  It’s in /etc/ too, in /etc/crontab.

There’s also our friend NTP, it has a configuration file there too.

The /etc/ directory was designed so that there wouldn’t be configuration files all over the system, instead, they’d be kept in one place.  This makes it much easier to manage the system, you don’t have to go searching in multiple directories to find the files.

Alpha:~ computerlamp$ cat /etc/hostname
Alpha

Yup, there’s the hostname of my computer.  Alpha.

Another file is /etc/motd, it contains the message of the day.   When you first open a terminal, you see the contents of this file. Sometimes it’s empty, sometimes it will have important messages, and sometimes it just has silly things:

Alpha:~ computerlamp$ cat /etc/motd
 _______
< Alpha >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Yes, that is cowsay in my /etc/motd file.  I was feeling whimsical.

The point of the /etc/ directory is to have the configuration files in one place, not scattered all over the computer.  It makes the system a lot easier to manage.

Plus, cowsay is fun!  All motd files should use cowsay!

The password file

Leave a reply

A while back I talked about how users have ids and how we can find out what they are using a command.  Remember, Linux loves files, so this information must be kept in a file somewhere.  It’s called the password file.  It should be called the ‘here are the users’ file, but it’s the password file.

You can actually look at this file.

Alpha:~ computerlamp$ more /etc/passwd

Here’s an example of what that can look like:

root:x:0:0:root:/root:/bin/tcsh
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:

There’s 7 different parts to each line of this file and they’re separated by a colon (:).  Each part is:

  1. username
  2. password
  3. user ID
  4. group ID
  5. real name
  6. home directory
  7. login shell

Let’s pull apart each part of this file.  We know what the user name is, mine’s computerlamp.  What’s yours?

The next one is the password.  It looks weird in that file, there’s an x where you’d think we’d see the encrypted password.  That’s because it isn’t in this file, that’s actually what we call a placeholder.  It’s a ‘one used to be here, but isn’t any more’.  That’s because the actual encrypted password is now in a different file, /etc/shadow that only root can read.

Then there’s the user ID and group ID, we know what those are too and I’d assume you know what your real name is, right?

The last two are the home directory and login shell.  That’s the directory you’re dropped in when you first log in and the shell that you’re going to use.  My home directory is /home/computerlamp and my shell is /bin/bash.

There’s something weird in that result though, right?  For example, the user mail has a login password that’s /sbin/nologin.  If someone tries to login to that account, then they’ll get a polite ‘no, you can’t do that’ message and will be logged out.  Another way of doing this is by putting /bin/false as the shell.

Daemons

Leave a reply

Linux® has a lot of processes that run in the background. This means that they don’t disturb you when they run, they sit back there and, well, run. For example, the syslog process, the process that sends email, init, the great granddaddy of processes, they all sit back there and do their job without bothering you. They’re actually all called daemons.

Daemon in the flesh

Daemon in the flesh

That little guy is kind of cute, isn’t he?

The name came from a thought experiment known as Maxwell’s demon. In this experiment, an imaginary daemon moved things around in the background. They also come from Greek mythology, where the daemon is a guy working in the background but who isn’t necessarily up to no good. He’s not necessarily up to good either, he’s just back there doing his thing. The Greek Mythology spelling is the one that stuck for the daemons on the system.  That little guy is the mascot of the BSD® operating system.

A general rule is that daemons on the system end their name with ‘d’ so you know they’re daemons. syslogd is the daemon for syslog, dhcpd is the daemon that runs dhcp, and named is the daemon that runs DNS. This isn’t always true though, one of the main programs that handle email is called sendmail and so is it’s daemon. There’s no d at the end of that one.

I ran apropos daemon to see what I’d get:

arpd (8) - userspace arp daemon.
authorized_keys (5) - OpenSSH SSH daemon
blkmapd (8) - pNFS block layout mapping daemon
cron (8) - daemon to execute scheduled commands (Vixie Cron)
daemon (1) - turns other processes into daemons
daemon.conf (5) - turns other processes into daemons
dbus-daemon (1) - Message bus daemon
epmd (1) - Erlang Port Mapper Daemonepmd [-d|-debug] [DbgExtra...] [-port No] [-daemon] [-relaxed_command_c...
faked (1) - daemon that remembers fake ownership/permissions of files manipulated by fakeroot processes.
faked-sysv (1) - daemon that remembers fake ownership/permissions of files manipulated by fakeroot processes.
faked-tcp (1) - daemon that remembers fake ownership/permissions of files manipulated by fakeroot processe

 

This isn’t all of the output, I just chose the first few lines. You should do it on your system to see what you’ll get but there’s a lot of results, aren’t there?

We’ll talk about some of these daemons in the future, because most of them are important to keep your Linux® system running happily.

dmesg

Leave a reply

Remember the Linux Kernel and Device Drivers?  They log like everything else.  They also write their log messages to a special file and there’s a command dmesg that lets you see what’s in the file.

The special file has a name and it’s the ring buffer.

The One Ring

The One Ring

This doesn’t mean you have a ring in your computer.  No hobbits are going to go on a quest to retrieve the one ring, sorry.  Go watch The Lord of the Rings to see that ring.

Instead, the ring buffer is a way of storing the messages.  Think of it like a pile of blocks.

A pile of blocks

A pile of blocks

We only want the pile of blocks to be four blocks high, so every time we add a block to the bottom, we must remove one of the blocks from the top.  That’s how the ring buffer works.  It only has a certain amount of information it can store so when that amount is reached, the oldest message in the ring buffer gets removed and the newest one is added.

Next question, what do the messages look like?  Well, you can run the command dmesg  and find out.  Some systems have that command restricted because the people who run the system think it’s a security leak.  It can be, if a device driver is complaining a lot a malicious person can take advantage of that.

I’m here to save the day, though, and show you what an output can look like:

___slab_alloc+0x37a/0x4a0
 ? __es_insert_extent+0x84/0x2f0
 ? ext4_ind_map_blocks+0x102/0xc70
 ? ext4_ind_map_blocks+0x584/0xc70
 __slab_alloc+0x27/0x40
 ? __slab_alloc+0x27/0x40
 ? __es_insert_extent+0x84/0x2f0
 kmem_cache_alloc+0xd5/0x180
 ? __es_insert_extent+0x84/0x2f0
 ? __es_insert_extent+0x84/0x2f0
 __es_insert_extent+0x84/0x2f0
 ext4_es_insert_extent+0xbf/0x190
 ext4_map_blocks+0x262/0x630
 ? __add_to_page_cache_locked+0xb0/0x170
 ext4_mpage_readpages+0x282/0xa20
 ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
 ? queue_unplugged+0x33/0xa0
 ? alloc_pages_current+0x58/0xb0
 ext4_readpages+0x29/0x40
 ? ext4_readpages+0x29/0x40
 ? ext4_journalled_zero_new_buffers+0x110/0x110
 __do_page_cache_readahead+0x131/0x1e0

That’s not very helpful, is it?  It requires a lot of knowledge to understand the output of dmesg and it’s usually used when the person running the system notes a problem.  When a system is restarted, the ring buffer is full of information about starting processes, so that can be useful then as well.

Syslog aka System Log

Leave a reply

In the last post we talked about how syslog is the central method for logging messages.  It really saves us from having to search all over the system for logs and it also lets the developer log messages without writing their own way of doing it.

Now for the big question, what does syslog do with all those messages?

That’s a good question, we still don’t want to be searching for those messages on our Linux® system.

Looking for the Logs in all the wrong places

Looking for the Logs in all the wrong places

Luckily for us, syslog has a configuration file that tells us everything we need to know.  That file is often in /etc/syslog.conf, but depending on your system, it might be somewhere else.  Check the man page for syslogd (that’s the actual program that does the logging) and it will tell you where the file is.  No matter where it’s located, it bas the same purpose.  To tell you where messages are put, based on the priority or the facility.

We know what the priority is, we saw that last time.  What’s the facility?  That’s the purpose of the program making the log message.  For example, if e-mail is logging, then the facility is mail.  If the Linux® kernel is logging, then the facility is kernel.

We know that emerg messages are ‘OH NO THE SYSTEM IS GOING DOWN’, so we probably want to see those on the screen, not shoved away to a log file.    The configuration that does that looks like:

*.emerg							/dev/console

What about mail messages?  Those we would like to put in a file.  It ‘s standard on most Linux® systems to put log messages in /var/log/.  So we’ll put all our messages from the mail system in one place.

mail.*                      /var/log/mail

You can get really complex with the configurations, you want to put mail.debug in one place, mail.alert somewhere else, then you can also put duplicates of everything in a third.  Or you can just stick them all in one place.  The person running the system usually decides this and often the decision is made to leave the defaults alone.  What are the defaults?  Those are the configurations that the system has when everything is first installed.

 

Logging

Leave a reply

I bet every time Iron Man does something in his suit, Jarvis makes a note of it.  ‘Shot a Chitauri out of the ski’, ‘did a flip’, ‘took Pepper on a flight’.  That way if something goes wrong, there’s a record of it.  In the computing world, this is called logging.

Making a note every time a computer does something interesting is a good thing.  If something breaks, you have notes that will hopefully let you figure out what happened.  If every program on your computer logged differently, that would be a nightmare, especially if they all put their logs in different places.  Troubleshooting would not be fun.

Instead, Linux® has a program that will do the logging for you.  You just tell syslog what you want to log and how important it is.  Then the syslog daemon will take care of everything for you.   The importance of a message, also known as the priority, has several defined meanings.

From the syslog man page, we can see the list:

  • LOG_EMERG system is unusable
  • LOG_ALERT action must be taken immediately
  • LOG_CRIT critical conditions
  • LOG_ERR error conditions
  • LOG_WARNING warning conditions
  • LOG_NOTICE normal, but significant, condition
  • LOG_INFO informational message
  • LOG_DEBUG debug-level message

A message logged with the level LOG_EMERG means ‘Danger!  Eek!  Fix me now!’ or ‘It’s too late, we’re going down Iron Man!’.  It can happen when the hard drive decides to fail dramatically.  The syslog can scream ‘Dead Hard drive ahead!’ then fall over dead.

That’s not exactly what happens, but it sounds fun!  My computer yelling ‘Dead Hard drive ahead!’ would certainly get my attention.

On the other end of the list is the LOG_DEBUG message.  Debug messages are usually used by developers, that is, the people who make the programs that run on your system.  The developers use them to debug, that is, find the bugs in their code.  These aren’t literal bugs, of course, but rather things that are unexpectedly happening that shouldn’t be.

Actual bug not found in computer

Actual bug not found in computer

Next time we’ll talk more about logging and syslog.

Linux® Scratch Paper

Leave a reply

Sometimes in algebra when I’d be working out a problem, I’d find myself grabbing a small piece of paper and working out a side problem.   Like ‘gee, what IS 9×19’?  I need that to solve the problem, but my teacher hated it when I made what she called messy marks on my test.  So, I had to use scratch paper.

I even find myself using scratch paper today, like when I’m on my phone and someone says, ‘write down this number!’.  Normally I’d save that in my phone, but my phone doesn’t actually let me both listen to the number and save it in the phone.

Linux® needs scratch paper sometimes too!  Sometimes a process needs to write to a file to say, ‘I did this’ or ‘I did that’.  It doesn’t really do that.  It writes state to a file so that if it has to stop it can pick up where it left off.   state means that ‘Hey, this is what I was going now, let me know if I need it later.’

This can get complicated.  Every process would need its own directory to write its own files, so imagine keeping track of all of them.

Linux® likes to keep things simple and straightforward.  Unix is the same way, and Linux® picked up this trick from them.  Instead of each process having its own directory, we have one central directory that everyone has write permission to.  It’s called the /tmp/ directory.  tmp stands for temporary, so it’s meant for files that aren’t supposed to stick around.  In fact, every time you reboot your computer, the /tmp/ directory is emptied.

/tmp/ can have lots of files in it or it can be empty.  In fact, when I look at /tmp/ on Alpha, I see:

Alpha:~ computerlamp$ ls /tmp/
Alpha:~ computerlamp$

That means in my case, the directory is empty.  What does it look like on your system?

Block vs Character: Cage Match

Leave a reply

In the last post, we looked at the contents of the file /proc/devices and saw a list of character devices and a list of block devices.  But what’s the difference between a block and a character?  And if there is a difference, when do we use a block vs character device?

Well, we know these are blocks:

Blocks

Blocks

And we know Ironman is certainly a character:

Ironman

Ironman

But that’s not what our devices do.  That would be silly if that’s what they did, right?  Our Linux® box isn’t Ironman and it isn’t a pile of blocks, it’s a computer that does things.

The difference between a block device and a character device is all in how your system communicates with it.  A character device is talked to one character at a time.  It doesn’t seem like it, but the terminal screen you type on is a character device.  One character at a time goes from your keyboard and to the screen (when you type) and one character at a time (although very quickly) goes from the output of the command you type and to the screen.

A block device sends its communications in blocks of data.  If you write information to a disk, like your USB drive or a hard drive, it goes in chunks of data.  For example, copying your file that contains all the Pokemons you’ve ever caught goes one big chunk at a time rather than P i k a c h u it sends it all together as Pikachu.

Now when we use Linux© we don’t care about block vs character.  We just care that what we type gets shown on the screen or what I copy to my USB drive goes to the drive.  How it really gets there isn’t important.

The people that really care about block vs character are the ones that have to write the code that makes the devices run.  That’s their job, but it is useful to know that there’s more than one type!

 

/proc and its files

Leave a reply

Last time we talked about how /proc has directories for each process but that there were also some files in that directory.  This time we’ll talk about a couple of those files.  /proc and its files have lots of information about the running of your system.

Let’s start with meminfo.  That’s a weird name, right?  Well, surprise, surprise, it has something to do with memory.  It has the current state of the memory on your system.

MemTotal:       32953488 kB
MemFree:        12891648 kB
Buffers:           56460 kB
Cached:          7736136 kB
SwapCached:       720288 kB
Active:         12263732 kB
Inactive:        5540668 kB
Active(anon):    9316432 kB
Inactive(anon):  3067828 kB
Active(file):    2947300 kB
Inactive(file):  2472840 kB
Unevictable:       26100 kB
Mlocked:           15880 kB
SwapTotal:       3117180 kB
SwapFree:        1736868 kB
Dirty:              1820 kB
Writeback:             0 kB
AnonPages:       9560840 kB
Mapped:          2414060 kB
Shmem:           2369068 kB
Slab:             768816 kB
SReclaimable:     521000 kB
SUnreclaim:       247816 kB
KernelStack:       18416 kB
PageTables:       365784 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    19593924 kB
Committed_AS:   27252100 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      276548 kB
VmallocChunk:   34359361512 kB
AnonHugePages:    491520 kB
DirectMap4k:        7488 kB
DirectMap2M:     4179968 kB
DirectMap1G:    29360128 kB

That’s a lot of information and we don’t really care what it all means.  The first two lines are the interesting part, they tell us the total memory on the system (in kilobytes). This system has 32 Gigabytes of memory installed on it and 39% of it is being used.  (I figured that out by dividing the amount of memory being used by the memory on the system).

Another file is the devices file.  That’s a list of devices on the system.  Remember, /dev/  has a list of all possible devices, the devices file in /proc has a list of all the devices on the system.  I can cat that one and see:

Character devices:
  1 mem
  2 pty
  3 ttyp
  4 /dev/vc/0
  4 tty
  5 /dev/tty
  5 /dev/console
  5 /dev/ptmx
  7 vcs
 10 misc
 13 input
108 ppp
128 ptm
136 pts
229 hvc
254 uio

Block devices:
  1 ramdisk
259 blkext
  7 loop
  9 md
 43 nbd
147 drbd
202 xvd
253 device-mapper
254 mdp

So the question now is, what’s a block device and what’s a character device?  Tune in next time for more information… Same bat time, same bat station!

/proc and process

Leave a reply

Everything is awesome! I mean, in Linux, everything is a file! We even saw where devices are files. But we also have processes, things that run and do things. Are they files? Well, the answer is… (drum roll)…yes. Everything is a file. Where devices are in /dev, process are in /proc. That kind of makes sense, right?

If you ls /, the top directory on your system, you’ll see something like:

bin   dev  home  lost+found  misc  net  opt   root  selinux  sys  usr
boot  etc  lib   media       mnt   od   proc  sbin  srv      tmp  var

So let’s cd into /proc and look around:

1     13     19     2705  2957   328  6          devices      kmsg          softirqs
10    14     2      2769  2972   329  6960       diskstats    kpagecount    stat
1013  14761  20     2771  2985   33   6962       dma          kpageflags    swaps
1043  14766  21     2791  2986   330  6963       drbd         loadavg       sys
1067  15     22     2794  2988   331  7          driver       locks         sysrq-trigger
1076  158    23     28    3      332  8          execdomains  mdstat        sysvipc
1095  16     24     2803  314    333  9          filesystems  meminfo       timer_list
11    160    24853  2841  315    334  901        fs           misc          tty
1102  162    25     2882  316    335  buddyinfo  interrupts   modules       uptime
1150  17     25077  29    320    336  bus        iomem        mounts        version
1152  171    25265  2901  32082  337  cmdline    ioports      net           vmallocinfo
1153  1743   255    2915  322    338  config.gz  irq          pagetypeinfo  vmstat
1161  1745   257    2932  325    339  consoles   kallsyms     partitions    zoneinfo
1173  1746   26     2940  326    34   cpuinfo    kcore        self
12    18     27     2949  327    5    crypto     key-users    slabinf

 

The numbers in proc correspond to processes that are currently running on your Linux system. On this system, there’s a process numbered 29 and one numbered 2901. There’s even 1150, 1152 and 1153. This system isn’t that busy, there’s only 98 processes currently running on it. That may seem like a lot, but a busy web server or a busy mail server could have a thousand busy processes.

Each one of those numbers is a directory and in that directory are files (see what I did there) that are information for that process. I looked in /proc/1 and saw:

ls: cannot read symbolic link 1/cwd: Permission denied
ls: cannot read symbolic link 1/root: Permission denied
ls: cannot read symbolic link 1/exe: Permission denied
attr/       coredump_filter  fdinfo/   mem         oom_adj        root@      status
auxv        cwd@             io        mountinfo   oom_score      sessionid  syscall
clear_refs  environ          limits    mounts      oom_score_adj  smaps      task/
cmdline     exe@             loginuid  mountstats  pagemap        stat       wchan
comm        fd/              maps      net/        personality    statm

What do those first three lines mean? Well, they mean I don’t really have permission to loo at everything in this directory. I could see that further if I did an <b>ls -l</b> on it, but this is really just to show you what’s in the directory.

Next time we’ll talk about some of the files in the /proc directory.