Category Archives: Internals

There’s another way

Back in this post I said ‘there’s more than one way to do it’ should be a motto of Linux©.  We’re going to do it again, this time with memory.  There’s another way to do it.

There’s the free command that shows you how much you have:

Alpha:~ computerlamp$ free
             total       used       free     shared    buffers     cached
Mem:        509688     491316      18372          0      97264     208964
-/+ buffers/cache:     185088     324600
Swap:       262140        316     261824

And we can give it a flag to show that information in gigabytes:

Alpha:~ computerlamp$ free -g
             total       used       free     shared    buffers     cached
Mem:             0          0          0          0          0          0
-/+ buffers/cache:          0          0
Swap:            0          0          0

Wait, something’s wrong.  There’s no memory on this system?  Well, it turns out that this one is an older Alpha.  (I logged into the wrong system this morning) and it doesn’t even have a gigabyte of memory.  I have to give it the -m flag.

Alpha:~ computerlamp$ free -m
             total       used       free     shared    buffers     cached
Mem:           497        479         18          0         94        204
-/+ buffers/cache:        180        317
Swap:          255          0        255

I really should have logged into the new Alpha and not the old one.

Anyway, back to the old Alpha.  There’s another way to look at memory, this time we use the command vmstat.

Alpha:~ computerlamp$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0    316  18488  97276 209068    0    0     0     0    1    1  0  0 100  0  0

The columns under the memory tag show us how much we have sitting around.

So, there’s another way.  Always another way in Linux©.

IP Address Origins

You know how to find your IP address, right? You can just run ifconfig and it will tell you.  Now the big question: How did your computer get that IP address? It didn’t buy one at the IP store or pull one out of the closet, it had to get it somehow. Well, there’s two places it
could have come from, and in this post we’ll talk about where your IP address came from.

The first choice is called a static assignment. That means your ISP picked an IP from the collection it has and assigned it to you.

Home address

Home address

It’s like the address of your house. It isn’t going to change, it’s always going to be that address. Unless you move your house completely, then your house address will change. If you move ISPs, then your IP address would change.

This used to be the way IP addresses were assigned to everyone. Remember how I said we’re running out of IP addresses? That doesn’t work as well any more. Now we have a dynamic pool.

Dead Pool

Dead Pool

Not Dead Pool, but a dynamic pool of IP addresses. When you tell your ISP ‘Hey, I’m going online’ it takes an IP address from its pool and assigns it to you. When you turn off your computer or your router, you give that IP address back to the pool.

Swimming Pool

Swimming Pool

Well, not into the swimming pool but rather the collection of IP addresses.

This is called DHCP, or Dynamic Host Configuration Protocol. That’s quite the mouthful, so we’ll stick with DHCP.

DHCP usually comes with a timer. That means you have to renew your IP address periodically. Normally it just means you say ‘yes, I’m here, I’m still using it, go away’ but depending on how the configuration works, you might get a new IP address. In other words, your IP addresses aren’t set in stone like your house address, but are likely to change. Like if you had a houseboat and moved it often, then that house address would change.

ifconfig output

In this post, I brought up the command ifconfig and I promised to talk about the output. It is rather confusing, so let’s pull it apart bit by bit so we can understand that ifconfig output.

eth0      Link encap:Ethernet  HWaddr 00:1A:2B:3C:00:00
          inet addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fd91:ae3b:d5e8:aa85:0:0:0:0 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:21601203 (20.6 MiB)  TX bytes:6145876 (5.8 MiB)
          Interrupt:21 Base address:0xe000

Let’s begin at the beginning! The first line says

eth0      Link encap:Ethernet  HWaddr 00:1A:2B:3C:00:00

That means that the link is an ethernet link. There is an ethernet cable attached to this computer connecting it to the Internet. The next part of the line says HWaddr and has some letters and numbers after it. We talked about IP addresses before and those can change on your computer. If you have a laptop and take it to a coffee shop, you’ll get a different IP address from the one you’ll have at home or that you would get in a library. Your computer never knows what IP address is going to show up, it isn’t like your home address which never changes.

The HWaddr is the fixed address of your computer, like your home address. It doesn’t change, no matter what. It’s also known as the MAC address. It’s set by the manufacturer of your ethernet card and never changes, no matter where you take your computer. You could take it to the Triskelion and it wouldn’t change.

Oh wait, that got destroyed in Captain America: Winter Soldier, didn’t it. You can’t take it there, but you could take it to the North Pole and it wouldn’t change.

The next line:

net addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0

Is information about your IP address and we already discussed that. Next time, we’ll talk about the line that starts with ip6. This ifconfig output is extensive, isn’t it? Lots of information that tell us all about the network interface on the computer.

 

Devices

Way back at the beginning I said that Linux® treats everything as a file.  Everything!  That includes devices, like the network interface, the wireless interface, the USB port, the motherboard, the sound board, speakers, microphones, you get the idea.  Every device you could add to your computer is treated like a file.

We can’t have those files floating around willy nilly, can we?  That’d be very confusing very quickly.  Sort of like asking the Hulk to keep track of Hawkeye’s arrows.  it would go really badly.

Hawkeye and His Arrows

Hawkeye

Instead, Linux® keeps all those files in one place, known as the /dev directory.  Every device has a file in this directory that Linux® uses to communicate with it.

If I do ls -l on /dev I get 676 lines.  I know because I did this:

Alpha:~ computerlamp$ ls -l /dev |  wc -l 
676

That doesn’t mean there are 676 devices on my system!  I’d have one busy system if that was the case.   It means that there are the possibilities for that many devices.

Let’s look at some of the output from ls -l:

Alpha:~ computerlamp$ ls -l /dev/
crw------- 1 root root  10, 235 Aug 11 02:19 autofs
crw------- 1 root root   5,   1 Aug 11 02:19 console
lrwxrwxrwx 1 root root       11 Aug 11 02:19 core -> /proc/kcore
crw------- 1 root root  10,  58 Aug 11 02:19 cpu_dma_latency
drwxr-xr-x 3 root root       60 Aug 11 02:19 disk
crw------- 1 root root  10,  63 Aug 11 02:19 dlm-control
crw------- 1 root root  10,  62 Aug 11 02:19 dlm-monitor
crw------- 1 root root  10,  61 Aug 11 02:19 dlm_plock
crw------- 1 root root  10,  60 Aug 11 02:19 ecryptfs
lrwxrwxrwx 1 root root       13 Aug 11 02:19 fd -> /proc/self/fd
crw-rw-rw- 1 root root   1,   7 Aug 11 02:19 full
crw------- 1 root root  10, 229 Aug 11 02:19 fuse

That isn’t very helpful, is it.  I don’t know what dlm means and why there’s a dlm-control or a dlm-monitor  I’m going to guess that the disk directory has something to do with hard drives, but I’m not sure.  encryptfs looks interesting, I bet it has something to do with encrypting things, but I’m not sure of that either.

The other thing to pay attention to is that the ownership and permissions of a lot of the files in this directory are restricted to root.  That means that while I can use ls -l to see that they’re there, I can’t actually look at them.

IPv6 addresses

IP addresses are formally called IPv4 addresses.  This post is to disuss the next generation of IP addresses, called IPv6 addresses.

This is kind of like Star Trek: The Next Generation, but without Data and Picard and the whole crew.

 Ok, it’s nothing like them.

In this post I said that there are 232 IP addresses, or 4,294,967,296 IP addresses.  Some of these are reserved and shouldn’t be used on the Internet, but that’s still a lot of IP addresses, right?  Enough to almost give every person in the world their very own IP address.  That leaves out companies having a whole bunch of computers on the Internet, or Universities with their computers, or even light bulbs with their IP addresses.  We’re running out of IP addresses!

The smart people that work on engineering the Internet, known as the IETF decided to solve this problem.  On a side note, they really could have come up with a better name.  Maybe Super Hardworking Internet Engineers Leading Development.  That sounds awesome, right?  They’d have to come up with their own Nick Fury though.

Nick Fury

Nick Fury

Back to the problem.   232 IP addresses aren’t enough.  The geniuses at the IETF decided not to just double that number, but to square it, and then square that number.  That gives us 2128 IP addresses, or 340,282,366,920,938,463,463,374,607,431,768,211,456 IP addresses.

Have fun reading that number out loud!

They also changed the notation for the IP addresses.  They don’t look like IPv4 addresses, they use : instead of . to separate numbers.  They also use base 16 instead of base 10 for the addresses.  Bet you didn’t think there would be math in this blog, did ya.

Here’s an example of an IPv6 address:

2601:547:902:cba7:4b8:ff43:f419:cd01

That’s a mouthful to remember.  I have an easier time remembering 192.168.0.1.

The IPv6 protocol that the SHIELD, okay, IETF engineers designed does a lot more than just creating the IP addresses, but that’s all we’re concerned with for right now.

Now remember that ifconfig output?

eth0      Link encap:Ethernet  HWaddr 00:1A:2B:3C:00:00
          inet addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fd91:ae3b:d5e8:aa85:0:0:0:0 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:21601203 (20.6 MiB)  TX bytes:6145876 (5.8 MiB)
          Interrupt:21 Base address:0xe000

The third line is the IPv6 address of the system.  A system can have both kinds on one interface.  Neat, huh.

More ifconfig!

We’ve talked about IPv6, we’ve talked about broadcast and other parts of the ifconfig output, but there’s more ifconfig output to look at.   The rest of the story, as it were.  So we have this output:

eth0      Link encap:Ethernet  HWaddr 00:1A:2B:3C:00:00
          inet addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fd91:ae3b:d5e8:aa85:0:0:0:0 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:21601203 (20.6 MiB)  TX bytes:6145876 (5.8 MiB)
          Interrupt:21 Base address:0xe000

eth0 is the interface for your computer to the internet.  That thing that all the traffic goes through.  That means that there’s Internet traffic going through that device.   ifconfig shows us more information than just the IP address.

Network Card 2 clip art

Let’s skip to these two lines:

          RX packets:41620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40231 errors:0 dropped:0 overruns:0 carrier:0

The Internet breaks all traffic up into packages called packets.  Think of it like a really really long text.   Every 160 characters in your text, your phone will send along the message and wait for the next bit.  (That 160 characters depends on your phone.  It could be more, it could be less.)  Your computer does the same thing.  It takes your messages and breaks them into chunks and sends it through the Interface.  It can also receive things through the interface.

The RX packets packages string in the ifconfig output is the number of packets your computer received and the TX packets is the number of packets your computer transmits.

Now these lines:

RX bytes:21601203 (20.6 MiB)  TX bytes:6145876 (5.8 MiB)

Are the total amount of traffic that went through all those pipes.  We received 20.6 megabytes of data and we sent out 5.8 megabytes of data.  We were busy surfing the web and pulled down the fun stuff.

The last bit of the output that we’re going to talk about is this line:

collisions:0 

A collision occurs when two devices on the same network try to transmit at the same time.  ifconfig keeps track of this because it usually means that there’s something wrong with the network and it should be looked at.

Now we’ve discussed all the important parts of the network interface!  If you want more information, read this site.

 

Interface

I talked about ports and services and now I’ll talk about the interface.  The interface is the actual network connection that has all that traffic to the ports come through it.  It’s that ethernet cable attached to your computer.

Ethernet Cable

Ethernet Cable

Or it’s that connection to the wireless network.  It’s the pipe (not a real pipe!) that connects your computer to the Internet.  Your computer has one, since you’re reading this blog post on the Internet.  Since we’re using Linux® there’s a way to see what interfaces you have.  It’s called ifconfig.

If you try to run ifconfig at the commandline, you’ll see:

Alpha:~ computerlamp$ ifconfig
-bash: ifconfig: command not found

That means it isn’t in your path.  (We’ll talk more about that later).  It just means that you have to give the full command line to see the output:

 

Alpha:~ computerlamp$ /sbin/ifconfig
lo        Link encap:Local Loopback
         inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:480 (480.0 b)  TX bytes:480 (480.0 b)

eth0      Link encap:Ethernet  HWaddr 00:1A:2B:3C:00:00
          inet addr:192.168.0.10  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fd91:ae3b:d5e8:aa85:0:0:0:0 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:41620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40231 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:21601203 (20.6 MiB)  TX bytes:6145876 (5.8 MiB)
          Interrupt:21 Base address:0xe000

You have at least two interfaces on your system, the first one is your local link.  That’s the interface that your computer talks to itself on.  It isn’t a real interface, you don’t plug a cable into it, but it’s the basic ‘I’m talking to myself!’  interface that every computer has.  The second interface is eth0, or ethernet 0.  If you have two ethernet interfaces (meaning you plug two ethernet cables into your computer) you’ll see eth0 and eth1.  And then things can get weirder.

Wireless interfaces don’t have a standard.  They can be named all sorts of things.

Next time, we’ll talk about what the actual output from ifconfig means.

ICMP

If the Avengers: Age of Ultron taught us anything, its that sometimes ideas fail spectacularly. Case in point, Ultron wanted to destroy the world to save that. That was a big mistake by Tony Stark, and not just a big mistake, a  BIG BIG mistake. On the Internet, sometimes things fail as well, just not as spectacularly as Ultron, but still, a failure. There’s a protocol to tell us that something went wrong, it’s called ICMP. It’s not nearly as dangerous as Ultron trying to make the world explode, and that’s a good thing.

Ultron

Ultron

In this post I said there were 65535 ports on a system. What happens if you try to access port 99999? That port doesn’t exist!

ERROR ERROR ERROR

ERROR ERROR

 

The good news is that your computer won’t blow up. That would be a good plot for a movie, don’t you think? Send a connection to a system on a particular port and watch the computer go BOOM. (Hollywood, if you use that, I thought of it first.)

The other good news is that you won’t even notice it when it happen, unless you’re sitting there watching every little bit of traffic coming to your computer (if so you need a new hobby). Your computer silently handles the problem by sending an ICMP packet saying ‘Nope, that’s not here.’

Your computer can also send ICMP packets out. The program ping uses what’s called an ICMP ECHO_REQUEST packet to see if another computer is alive and to find out how long it takes to get there. It runs like:

 

Alpha:~ computerlamp$ ping www.google.com
PING www.google.com (173.194.205.103): 56 data bytes
64 bytes from 173.194.205.103: icmp_seq=0 ttl=45 time=27.505 ms
64 bytes from 173.194.205.103: icmp_seq=1 ttl=45 time=24.952 ms
64 bytes from 173.194.205.103: icmp_seq=2 ttl=45 time=23.545 ms
64 bytes from 173.194.205.103: icmp_seq=3 ttl=45 time=24.521 ms
64 bytes from 173.194.205.103: icmp_seq=4 ttl=45 time=23.499 ms
64 bytes from 173.194.205.103: icmp_seq=5 ttl=45 time=27.609 ms

It keeps going and going and going… until you use Control-C to stop it.

When I do hit control-C, it tells me some statistics about the traffic:

--- www.google.com ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 23.499/25.272/27.609/1.695 ms

Or, I can use the -c flag for ping. That way it will only do the number of pings I tell it to:

Alpha:~ computerlamp$ ping www.google.com
PING www.google.com (173.194.205.103): 56 data bytes
64 bytes from 173.194.205.103: icmp_seq=0 ttl=45 time=24.166 ms
64 bytes from 173.194.205.103: icmp_seq=1 ttl=45 time=23.373 ms
64 bytes from 173.194.205.103: icmp_seq=2 ttl=45 time=23.462 ms
64 bytes from 173.194.205.103: icmp_seq=3 ttl=45 time=23.437 ms
64 bytes from 173.194.205.103: icmp_seq=4 ttl=45 time=23.783 ms

--- www.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 23.373/23.644/24.166/0.297 ms

 

Services

Remember the post where I mentioned /etc/services? Let’s talk more about that file.  It’s a list of all possible services you could run on your Linux® system, it doesn’t mean that your system is running those.

What port do web services use?  Well, web services are formally known as http, or HyperText Transport Protocol.  We’ll just call it http.   We can grep for http in /etc/services by:

Alpha:~ computerlamp$ grep http /etc/services

I got 105 lines back from that simple command.  I know that because I did this command when the first one scrolled a lot:

Alpha:~ computerlamp$ grep http /etc/services | wc -l

That’s a lot of ports.  Let’s pipe that command thorugh less and see what we get:

Alpha:~ computerlamp$ grep http /etc/services | less
#       http://www.iana.org/assignments/port-numbers
http             80/udp     www www-http # World Wide Web HTTP
http             80/tcp     www www-http # World Wide Web HTTP
http-mgmt       280/udp     # http-mgmt
http-mgmt       280/tcp     # http-mgmt
https           443/udp     # http protocol over TLS/SSL
https           443/tcp     # http protocol over TLS/SSL
gss-http        488/udp     # gss-http
gss-http        488/tcp     # gss-http
http-alt        591/udp     # FileMaker, Inc. - HTTP Alternate (see Port 80)
http-alt        591/tcp     # FileMaker, Inc. - HTTP Alternate (see Port 80)
http-rpc-epmap  593/udp     # HTTP RPC Ep Map
http-rpc-epmap  593/tcp     # HTTP RPC Ep Map
multiling-http  777/udp     # Multiling HTTP
multiling-http  777/tcp     # Multiling HTTP
netconfsoaphttp 832/tcp     # NETCONF for SOAP over HTTPS
netconfsoaphttp 832/udp     # NETCONF for SOAP over HTTPS
llsurfup-http   1183/udp    # LL Surfup HTTP
llsurfup-http   1183/tcp    # LL Surfup HTTP
llsurfup-https  1184/udp    # LL Surfup HTTPS
llsurfup-https  1184/tcp    # LL Surfup HTTPS
compaq-https    2381/udp    # Compaq HTTPS

The first line is a comment and that points us to the website that has all the registered ports.

The second line is the UDP port for HTTP, which is port 80.  TCP uses the same port.  That’s the basic port that all web traffic goes over, unless it is encrypted.  If it’s encrypted, it uses port 443.  That’s what this sentence means ‘http protocol over TLS/SSL’, it means ‘http traffic that’s encrypted’.

 

How TCP Really Works

I talked about TCP and UDP in this post. Now we’re going to talk about how TCP really works, how it keeps that connection going. It’s how mail works, it’s how a lot of web traffic works, so how does that pipe keep going?

 

A pipe

A pipe

Well, it isn’t a physical pipe like that picture. It isn’t a physical connection at all and it also isn’t a psychic connection, instead, it’s a virtual connection.

It starts by the two systems, let’s call them Captain America and Black Panther, negotiating a connection.  The connection is called a handshake.

A handshake

A handshake

Actually it’s called a three-way handshake and it works like this:

Captain America tells Black Panther, “Hey, I want to talk to you.”. He does this by sending a packet called a SYN packet.

Black Panther then tells Captain America, “Gotcha! Good idea, Let’s talk.” He does this by sending back a packet called an ACK. The three-way is because Captain America acknowledges Black Panther’s agreement to talk by sending back his own ACK.

The TCP connection uses these three packets to set the connection up, SYN, ACK and ACK.

So they’re talking away, but like in all conversations, sometimes there’s a lull where neither side says anything. Maybe they’re each getting a nice drink, maybe they’re each writing a note to themselves to make fun of Iron Man, maybe they’re just taking a nap. The connection sends back and forth KEEP ALIVE packets to say ‘Hey, you still there?’ and the other side acknowledges ‘Yup, still here. Taking a nap, be back to talking with you in a minute.’

All conversations eventually end, or at least take enough of a break that you hang up the phone and do something else for a while. When this happens, the TCP connection is shutdown, like when you hang up your phone. Your phone tells the other guy ‘yup, I’m done’ and shuts down the communication. Similarly, the one end of the TCP connection will tell the other ‘I’m outta here’, which ends the connection.