Category Archives: OSS

JQ 101

JQ is an amazing tool that allows you to manipulate JSON via the command line / bash.

To be frank I was a bit weirded out by people praising something so ‘trivial’ back in the day, but then I realized how powerful this thing actually is. Manipulating JSON coming from web background may seem really simple task, but when you go to bash, it is … nightmare.

JQ makes things doable in surprisingly simple way.

The thing it is that it can be a bit hard to start with. Some people have asked me to provide them with several examples so they can have something to build on top of when developing their own stuff.

These are the basics, if you have mastered everything listed here, your logical next step would be to read the official docs like a real dev

So, this is my cheat sheet:

How to use

Invocation

You can cat a file and pipe it to JQ like:

cat file.json | jq 'something'

but it is generally preferred to do it like this:

jq 'something' < file.json

I personally prefer to add -r flag to JQ so it outputs raw

jq  -r 'something' < file.json

Sample data

We are going to be working with this JSON for most of the examples

{
    "data": [
        {
            "email": "user1@example.com",
            "username": "user1"
        },
        {
            "email": "user2@example.com",
            "username": "user2"
        }
    ],
    "users": [
        {
            "email": "user1@example.com",
            "user.name": "user1"
        },
        {
            "email": "user2@example.com",
            "user.name": "user2"
        }
    ]
}

Examples

Basic

Simple select everything

jq -r '.' < file.json

will give you the root of the document. Doesn’t matter if it is an object or array

Select property of an object

jq -r '.data' < file.json

returns an array containing all users in ‘.data’

Select first user in data

jq -r '.data[0]' < file.json

returns first object in ‘data’ array

Select all usernames is ‘.data’

jq -r '.data[].username' < file.json

returns

user1
user2

Select all usernames of ‘users’ array when keys contain dots

jq -r '.users[]["user.name"]' < file.json

returns

user1
user2

Selectors

Getting keys of an object

jq -r '.|keys' < file.json

gives you an array containing all keys ["data", "users"]

Getting keys of object on new lines

jq -r '.|keys[]' < file.json

returns

data
users

Select only users called ‘user1’ from ‘data’

jq -r '.data[] | select(.username="user1")'

gives you {"username": "user1", "email": "user1@example.com"}

Select the emails of all users called ‘user1’

jq -r '.data[] | select(.username="user1") | .email'

Create JSON

Create object with key username and value ‘user1’

jq -n --arg mykey username --arg myval user1 '{($mykey):$myval}'

Append property to object

jq -n --arg mykey username --arg myval user1 '{($mykey):$myval}' | jq -r '. |= . + {"email": "user1@example.com"}'

or simpler:

echo '{"username": "user1"}' | jq -r '. |= . + {"email": "user1@example.com"}'

both result in:

{
  "username": "user1",
  "email": "user1@example.com"
}

Change value of object property

jq -n '{ "username": "user1"}' | jq -r '.username="user11"'

Create array

jq -n '[]'

same as

echo '[]' | jq -r '.'

Set value of array element

jq -n '[1,2,3]' | jq -r '.[2] |=  5'

outputs [1,2,5]

Add element to array (of unknown length)

jq -n '[1,2,3]' | jq -r '.[.|length] |=  4'

result [1,2,3,4]

PGP for the masses

Intro

Using PGP is very simple, you need to generate key (it will actually be two keys – public and private). Public key is what you want to give to other people so they can encrypt files for you. Private key is something you do not want to share.

This guide/cheatsheet assumes that you are using osx, but aside from the GUI tools, the rest will be the same for all operating systems

There are 4 things that you need to know:

  • how to generate your key
  • how to find other people’s keys
  • how to encrypt
  • how to decrypt

Terminology

Some clarifications:

OpenPGP is a standard for encryption

PGP stands for Pretty Good Privacy – this is software that implements the OpenPGP standard

GPG stands for GNU Privacy Guard – open source implementation of OpenPGP (this is what we are going to use)

PHP is something completely different

Install

Download from https://gpgtools.org/ If you are not planning to use it with Mail App, during installation click Customise and deselect GPG Mail

Create new key

You can just type
gpg --full-generate-key

and answer the questions or via GUI:
Click New, enter your e-mail address and password, wait for key generation to finish. At the end click to upload the key to key server.

If you have already generated key you can upload it by clicking Key -> Send Public Key to Key Server

Note: By uploading your key it will be uploaded to one of the SKS servers used by OpenPGP and it might take some time until it is visible in all servers.

Retrieving other’s public keys

If you have tin foil hat, you would prefer to ask someone to send you their PGP key and they can send it to you. Never trust keys that you have not asked for and you are not sure are sent by the person you think sits behind them, somebody may be pretending to be a friend.

If you however think that you know what you are doing, click Lookup key, type the e-mail of the person you expect to have published, check if the rest of the data looks credible and import it.

If you do not want to use the GUI app, you can search in websites like http://keys.gnupg.net

Note: Some people may have several keys, ask them which one you should use (usually the one that was created most recently would be the one that you want).

Note2: After publishing key it may take some time until it shows up, be patient

Encrypting files

Using Finder

Go to the folder where the file is, right click on it, select Services -> Open PGP : Encrypt File, then select the key that you want to be able to decrypt it and click Encrypt. If the file is called file.txt you will see a new file called file.txt.gpg

Using CLI

gpg -e --armor file.txt

Enter the e-mail of the recipient/s and hit enter when you are done

or if you want to encrypt some text:

echo "text" | gpg -ea

Decrypting files

Using Finder

Go to the folder where the file is, right click on it, select Services -> Open PGP : Encrypt File, enter your password and decrypted file should show up

Using CLI

gpg -d file.txt.gpg

View Keys

gpg -k

Adding PGP key in GitHub

Go to github.com -> Settings -> SSH and GPG keys -> New GPG Key

Copy your public key

gpg --export --armor MYKEYID

Paste it in github and click Save.

Then you have to tell git to sign your commits

git config --global commit.gpgsign true
git config --global user.signingkey MYKEYID

If you have created your github account with different e-mail address than the one specified in the PGP file, you can go to Settings -> Emails and add new e-mail address. Then proceed with the verification clicking the link sent to your inbox.

If you still have problems, check the e-mail specified in your ~/.gitconfig file. It should match the one from the PGP file

Deep(ish) dive

View secret keys with their long IDs

gpg --list-secret-keys --keyid-format LONG

Backup and restore secret keys

Backup:

gpg --export-secret-keys MYKEYID > my-private-key.asc

Import from backup:

gpg --import my-private-key.asc

Revocation

If you private key leaks or maybe you do not want to use it anymore you have to send the Key Server revocation certificate. Ideally you would still have the private key and you could generate it at any moment, but sometimes keys get lost, so you may want to generate revocation certificate now and back it up somewhere.

gpg --output revocation.crt --gen-revoke myemail@example.com

When you decide to revoke the key you have to import it and send it to the Key Server like this:

gpg --import revocation.crt
gpg --send-keys KEYID

Replace KEYID with the ID of your key

Signing other people’s keys

TLDR: You can sign keys to tell others that this key is good.

If you trust the person you can sign their key and send them back the key and you will be listed as someone who signed the key.

gpg --sign-key email@example.com

You have to send them the signed key back

gpg --output /tmp/signed.key --export --armor email@example.com

and they have to import it to benefit from you signing it:

gpg --import signed.key

If people you trust have signed the key, usually it is a good indicator.

Запазване на пароли под Linux, Windows и Mac

Много харесвам Keychain-a на Mac, но той си има своите ограничения – измежду които основното, че не можеш да го ползваш на други операционни системи.

Оказа се, че може би единствения мултиплатформен начин да направите това с KeePass. Има много начини и комбинации от софтуер, с които да имплементирате нещо такова, но ето какво аз съм намерил за най-доброто.

Под Debian KeePass2 присъства сред пакетите, така че можем да го инсталираме лесно

Следва да сложим plugin на Firefox, аз лично предпочитам PassIFox. Този плъгин изисква KeePassHttp за да работи, а на плъгина му трябва mono-complete.

След като инсталирате mono-complete, изтегляте KeePassHttp.plgx , слагате го  в /usr/lib/keepass2 и да рестартирате keepass.

На този етап паролите ви от Firefox се пазят в KeePass. Черешката на тортата е да сложите kdbx файла в OwnCloud за да имате синхронизация между всичите си устройства.

 

За Max Os въпреки че не е много стабилен препоръчвам  MacPass.

За Android също има owncloud и можете да ползвате KeePassDroid.

Дори iOS феновете ще останат приятно изненадани от комбинацията ownCloud + KeePass Touch.  За Windows мисля няма нужда да обяснявам.

 

Като цяло имате password management за поне 5 операционни системи, напълно безплатен и чрез ownCloud синхронизиран онлайн, най-близкото което можете да получите до Mac Keychain с бонуса, че работи на (почти) всички устройства, които вероятно притежавате.

 

 

Edit: November 2018

За Firefox Quantum PassIFox не работи. Вместо това можете да ползвате Kee заедно с KeePassRPC

Инсталиране на MiniDLNA под FreeBSD 10

Ако предпочитате да си компилирате всичко, порта се намира в net/minidlna. Тоест нещо от типа на:

Ако сте по-мързеливи винаги може да инсталирате с

Конфигурационния файл се намира в /usr/local/etc/minidlna.conf и нещата, които трябва да промените или разкоментирате  са:

Разбира се, направете нужните корекции за вашата система. Ако нямате папката може би е добра идея да я създадете:

По подразбиране папката където се намират мултимедийните файлове е в /opt. Трябва да създадете папката ако не съществува при вас. Опитах да използвам папка от типа /home/user/Video, но има проблеми с правата, които така и не разреших за това предпочетох да използвам симлинкове.

Ако всичко ви изглежда ок можем да стартираме услугата.

За да проверите дали всичко е ок можете да проверите log файла:

Ако няма грешки можете да проверите дали всичко работи през вашия телевизор или VLC.

За финал добавете minidlna към услугите, които да се стартират при пускане на машината като добавите към /etc/rc.conf

Накратко това е.  Доста подобно на това как се инсталира MiniDLNA под Mac OS X (за което писах преди), но с особеностите на freebsd.