Lynne Teaches Tech: What are protocols and formats? How do they come to be?

This question was originally submitted via the survey.

This question, like our last reader-submitted question, is in three parts:

• What exactly is a protocol?
• What exactly is a format?
• Can you give a short rundown of how standardisation in IT works?

I’ll answer them from top to bottom.

Protocols

Update 27/05/19: Corrected a statement about binary files being unreadable by humans.

Protocols are used to facilitate communication between computers. You need rules for what things mean and how to behave, otherwise your computer won’t know what to do with all the data it’s receiving.

For the sake of the explanation, we’ll create our own fake protocol, ETP: Face Transfer Protocol. You can use this protocol to send an emoji from one networked computer to another.

A proper ETP communication will have two pieces of information: An emoji, and a date the emoji was sent. When you connect your ETP client to an ETP server, you might receive this:

Emoji: 🍔
Date: 2019-05-23 10:31:45

Your ETP client can then format this raw data into something more readable.

🍔

Received on May 23rd at 10:31 AM from etp://bune.city

This isn’t a very useful protocol, but it gets the point across: A protocol allows computers to communicate using a predefined set of rules.

The client can treat this ETP data however it wants. It can show just the emoji, or just the date, or maybe it tells your printer to make 100 copies of the emoji. The ETP server doesn’t tell the client what to do with the data, it just sends the data. Your client might even send a response back.

A real world example is HyperText Transfer Protocol, or HTTP. To view this blog post, your computer sent a HTTP request, specifying that you want to view this particular page. The server then sent the page back, including some additional info, such as the size of the response, the encoding (such as text/html), the date it was last modified, and so on. The only information you end up seeing is the content – your web browser handled the rest.

HTTPS works the same way as HTTP in this regard. It’s the same protocol with encryption added, and otherwise behaves the same.

Formats

Formats are ways to store types of data on a computer. There are formats for documents (.docx, .pdf, .odt…), audio (.mp3, .ogg, .flac…) and so on. Documents are sort of similar to protocols in that they describe data to be interpreted. The main difference is that protocols work over a network, and formats are files. You can transfer files between computers using protocols, but you can’t use a format to transfer a file. You also can’t store a protocol as a file, because protocols are conversational – your computer needs to be able to talk to a HTTP file. You can, however, run a server on your computer to make and respond to HTTP requests.

Let’s invent a simple format, a .face file. A .face file looks like this:

>>>Face file version 1<<<
:)

The first line lets the face file reader know that it’s definitely reading a face file. The second line is the actual face. The first line appears in every single face file, and the second line changes between files. Another example would be:

>>>Face file version 1<<<
:(

While some files are easily readable by opening them with a text editor, such as with our .face file, others are unreadable messes of binary information. To read these types of files, a file viewer is always needed. While our face file can be understood without a viewer, we’re still going to create a viewer to demonstrate how information can be made to look nicer and more human-readable with the assistance of a program. Having a computer parse and display the information in a nicer format can make viewing a file a much better experience.

Let’s call our face file viewer FacesTime, because I’m uncreative. When you tell FacesTime to open a file, it first checks to make sure the first line is there. If it is, it assumes it’s looking at a genuine face file, and reads on.

It then gets to the second line, which is the data portion. FacesTime takes this previously unreadable :) and converts it into a human readable version, as demonstrated by the highly realistic mockup below.

Some formats are more flexible than others. Our .face format could be displayed as an emoji, or a drawing, or even a photo. When you have something like an audio file, however, there’s not much room for interpretation. You can change things like contrast and colour balance, but you obviously want the actual sound to remain unchanged. On the other hand, you could have a program that opens the file and displays a visualised version instead of playing the song, or a program that lets you edit the song title and album art without actually playing it. You can do what you want with formats, but most of the time, you just want to display the content of it in a human-readable (or listenable!) way.

A .txt file specifies nothing but the raw text inside it, meaning that you’re free to interpret it however you want. Your viewer can choose the font size, colour, and so on. However, a Word document is a different story. You could, of course, create a Word document viewer that always displayed text as Comic Sans, but then you wouldn’t have a very good document viewer. It’s usually best to just follow the standard, because everyone else is following it, too.

How are standards created?

Anyone can create a standard. I created two just then: The ETP protocol, and the .face format. I didn’t really define them very well, but I did define them to some extent. Someone could create an ETP server right now with the information provided. However, what actually happens with my standards is beyond my control. Someone could create an ETP server that allows for more than one emoji at a time. A text format could have a standard saying “all text must be displayed as hot pink on neon green”, but nobody actually has to follow that. It’s different if there’s a certification process, however.

To display a DVD Video label, a DVD player must comply with the DVD standard. If it does anything differently (such as playing all the video upside down), it doesn’t get approved, and can’t display the label. Again, there’s nothing stopping you from creating such a player, and if you don’t care about having the DVD Video label applied to your product (or being allowed to call your product a “DVD player”), then you’re good to go.

In the end, standards are just a guideline. Things will typically go badly if you don’t follow them, but you don’t have to, and sometimes you might want to violate them on purpose. An example is our .face format – you might want to violate the standard by including multiple faces. However, any standard .face file viewer wouldn’t be able to understand your modified version, so unless everyone agrees on your standard, then things probably won’t work out too well.

Summary

A protocol defines a method of communication for two computers to “talk” to each other. A format defines a method for one computer to read and display data. Standards can be created by anyone, and are occasionally enforced.

Thanks for reading!

Emoji: 😀
Date: 2019-05-24 00:45:18 AEST