GitHub Project of the Week: RetDec

Avast has open-sourced its machine code decompiler and analytical tool, RetDec, in a move to help the cybersecurity community fight against malicious software. The tool enables you to study application code without actually having to run the application.

“Decompilers can be used in a variety of situations,” wrote the Avast Threat Intelligence Team in a post. “The most obvious is reverse engineering when searching for bugs, vulnerabilities, or analyzing malicious software. Decompilation can also be used to retrieve lost source code when comparing two executables, or to verify that a compiled program does exactly what is written in its source code.”

The company open-sourced the tool because it wants to have a generic tool that is not limited to a single platform for analyzing code.

“By preserving a program’s functionality, we want the source code to reflect what the input program does as accurately as possible; otherwise, we risk assuming the program does one thing, when it really does another,” the team wrote.

RetDec has been in development since 2011, when it was created as a joint project by AVG Technologies and the Faculty of Information Technology of the Brno University of Technology in the Czech Republic. AVG was acquired by Avast in 2016, so it continued to work on finishing the compiler.

Features include new support file formats and architectures, static analysis of executable files, compiler and packer detection, loading and instruction decoding, signature-based removal of statically linked library code, and more. For a complete list of features, see here.

 

Developers can try out the decompiler in their browser using Avast’s web service. It can also be accessed using the REST API.

Top 5 trending projects on GitHub this week:

  1. 30 seconds of code: Understand these JavaScript snippets in 30 seconds or less!
  2. parcel: Blazing fast, zero configuration web application bundler
  3. Turi Create: A project designed to simplify the development of custom machine learning models
  4. Coding Interview University: Study to become a software engineer
  5. muuri: Responsive, sortable, filterable and draggable grid layouts
GitHub project of the week – ksonnet

Heptio is marking a year in business with the release of what it’s calling a “new chapter” for ksonnet, its tool for storing and generating application configurations for Kubernetes deployments.

Ksonnet uses a jsonnet, Heptio’s custom language for manipulating and producing JSON documents to generate Kubernetes manifests or YAML, which aren’t human-readable.

“Our first efforts with ksonnet were focused around creating patterns to help organize the configuration across many applications for large deployments of Kubernetes,” Heptio co-founder Joe Beda wrote in a blog post. “While Jsonnet/ksonnet work well in those situations, the language and concepts can be intimidating for new and casual users. We are fixing that today.”

The project’s readme on GitHub breaks down its function in five ways.

  • Reuse common manifest patterns (within your app or from external libraries)
  • Customize manifests directly with powerful object concatenation syntax
  • Deploy app manifests to multiple environments
  • Diff across environments to compare two running versions of your app
  • Track the entire state of your app configuration in version controllable files

The project’s authors at Heptio say that there’s catch-all solution for reproducing the configurations from one environment in another, but Beda writes that “we hope that ksonnet will be a good solution for many people.” He highlights Bitnami’s “less opinionated” solution, kubecfg, as something that’s helped inform Heptio’s development.

Next up for ksonnet is the ability to write Helm charts for the Helm 3 Kubernetes package manager.

Read more about the update to ksonnet and watch a video demonstration in the announcement.

 

GitHub Project of the Week: Infer’s RacerD

Facebook has announced the open source release of their RacerD unsynchronized memory access — or data race — detector for Java applications following its successful implementation internally.

RaceD is based off of the company’s open source static analysis tool, Infer.

According to Facebook, this newest publicly available piece of Infer’s static analysis platform has caught more than 1,000 multi-threading issues in their Android codebase over the past 10 months, all before the code ever reached production.

Though the project was started in 2015 with a lofty goal of creating an unobtrusive, “high speed and low friction,” scalable concurrency analysis utility for programmers writing concurrent programs, an immediate need caused the team to shift towards a tool that analyzed existing code. This development helped Facebook’s engineers on their Android app convert part of the News Feed from a sequential, single-threaded to a concurrent, multi-threaded operation, Facebook’s software engineer Sam Blackshear and software evangelist Peter O’Hearn writes.

“Data races are one of the most basic forms of concurrency error, and removing them can help simplify the mental task of understanding a concurrent program,” wrote Blackshear and O’Hearn.

The three guiding ideas behind RacerD’s design are:

  1. Don’t do whole-program analysis; be compositional.
  2. Don’t explore interleavings; track lock and thread information.
  3. Don’t attempt a general, precise alias analysis; use an aggressive ownership analysis for anti-aliasing of allocated resources.

“Reasoning about concurrency has been studied for over 40 years and has seen many research advances. However, not much of the work has made it through to deployment where it can help programmers in their daily jobs,” Blackshear and O’Hearn wrote. “RacerD demonstrates that a static concurrency analysis can be developed and effectively applied at the speed and scale demanded by Facebook’s development model, where a large codebase is undergoing frequent modification.”

Top 5 trending projects on GitHub this week:

#1. p3cAlibaba Java coding guidelines.
#2. napajsMicrosoft’s multi-threaded JavaScript runtime.
#3. Front end checklistLike the name says, a checklist for front end development.
#4. Tech Interview HandbookA cheatsheet for rocking your next interview.
#5. Design blocksMore than 170 Bootstrap based design blocks.

GitHub Universe outlines plans for the future of software development

About ten years ago, GitHub embarked on a journey to create a platform that brought together the world’s largest developer community. Now that the company believes it has reached its initial goals, it is looking to the future with plans to expand the ecosystem and transform the way developers code through new tools and data.

“Development hasn’t had that much innovation arguably in the past 20 years. Today, we finally get to talk about what we think is the next 20 years, and that is development that is fundamentally different and driven by data,” said Miju Han, engineering manager of data science at GitHub.

The company announced new tools at its GitHub Universe conference in San Francisco that leverages its community data to protect developer code, provide greater security, and enhance the GitHub experience.

“It is clear that security is desperately needed for all of our users, open source and businesses alike. Everyone using GitHub needs security. We heard from our first open source survey this year that open source users view security and stability above all else, but at the same time we see that not everyone has the bandwidth to have a security team,” said Han.

GitHub is leveraging its data to help developers manage the complexity of dependencies in their code with the newly announced dependency graph. The dependency graph enables developers to easily keep track of their packages and applications without leaving their repository. It currently supports Ruby and JavaScript, with plans to add Python support in the near future.

In addition, the company revealed new security alerts that will use human data and machine learning to track when dependencies are associated with public security vulnerabilities, and recommend a security fix for it.

“This is one of the first times where we are going from hosting code to saying this is how it could be better, this is how it could be different,” said Han.

On the GitHub experience side, the company announced the ability to discover new projects with news feed and explore capabilities. “We want people to dig deeper into their interests and learn more, which is one of the core things it means to be a developer,” said Han.

The new news feed capabilities allows users to discover repositories right from their dashboard, and gain recommendations on open source projects to explore. The recommendations will be based off of people users are following, their starred repositories, and popular GitHub project.

“You’re in control of the recommendations you see: Want to contribute to more Python projects? Star projects like Django or pandas, follow their maintainers, and you’ll find similar projects in your feed. The ‘Browse activity’ feed in your dashboard will continue to bring you the latest updates directly from repositories you star and people you follow,” the company wrote in a blog.

The “Explore” experience has been completely redesigned to connect users with curated collections, topics, and resources so they can dig into a specific interest like machine learning or data protection, according to Han.

Han went on to explain that the newly announced features are just the beginning of how the company plans to take code, make it better, and create an ecosystem that helps developers move forward.

“These experiences are a first step in using insights to complement your workflow with opportunities and recommendations, but there’s so much more to come. With a little help from GitHub data, we hope to help you find work you’re interested in, write better code, fix bugs faster, and make your GitHub experience totally unique to you,” the company wrote.

Introducing Atom-IDE

GitHub, in collaboration with Facebook, are pleased to announce the launch of Atom-IDE – a set of optional packages to bring IDE-like functionality to Atom.

The start of this journey includes smarter context-aware auto-completion as well as a host of code navigation features such as an outline view, go to definition, find all references as well as other useful functions such as hover-to-reveal information, errors and warnings (diagnostics) and document formatting.

Our initial release includes packages for TypeScript, Flow, JavaScript, Java, C# and PHP that utilize the power of language servers to provide deep syntactical analysis of your code and projects. The language server protocol is being adopted by a number of organizations including Microsoft, Eclipse, Sourcegraph, Palantir, Red Hat, Facebook and now GitHub too!

Get started

 

We strongly recommended you use Atom Beta 1.21 as it includes the necessary file monitoring and process control to ensure the underlying language servers are running properly.

You’ll need to install at least two packages – the user interface for Atom IDE and a package that support the language you wish to use:

  1. Bring up Atom’s Install Packages dialog (Settings View: Install Packages and Themes)
  2. Search for and install the atom-ide-ui package to bring in the IDE user interface
  3. Install the IDE language support you need (e.g. ide-typescript) – a summary of the ones available at launch include:

TypeScript & JavaScript (ide-typescript)

 

The ide-typescript package takes advantage of the Microsoft TypeScript server wrapped up in a language server protocol thanks to the work of the team at SourceGraph. While targeted at TypeScript it also works great with JavaScript providing you with autocompletion, document outlines, diagnostics and errors, etc.

Flow (ide-flowtype)

Our good friends over at Facebook have published ide-flowtype to bring the power of the Flow type annotation system to Atom.

C# (ide-csharp)

One of the earliest examples of a language server was OmniSharp for the C# language. By taking advantage of the node-omnisharp package we are able to bring many IDE-like features into Atom for C# via ide-csharp.

Java (ide-java)

The Eclipse foundation and Red Hat have been a big proponent of language servers and the Java package shows! You will need a Java 8 runtime installed to get going but then can enjoy much richer editor facilities. Check out ide-java.

PHP (ide-php)

The ide-php utilizes a PHP language server by FelixFBecker to provide support for the PHP scripting language. (Requires the PHP 7 runtime installed.)

Using Atom-IDE

Each of the IDE packages expose a selection of functionality that is dependent on the underlying language server and is activated when you open files it supports. (Some take a few seconds to start-up and others like ide-java and ide-php will take a short while on first open to download the language server itself.)

Here’s a quick summary of how these features are exposed within Atom IDE:

Autocomplete

Autocomplete is enabled in all the ide packages we are shipping today. Start typing to get improved results. Some providers may require you to manually trigger autocompletion by pressing CtrlSpace for performance reasons.

Diagnostics

You can see diagnostics by clicking the red exclamation mark and yellow warning triangle at the bottom left of your Atom window. This will open the new Diagnostics pane that shows you the errors and warnings and allow you to click them to jump right to that location in the code. You will also see indicators to the left of line numbers in the editor itself.

Find all references

Position the text cursor in the class or variable you are interested in then activate Find all references either from the right-mouse button menu or Find References: Activate from the command palette.

Some providers allow you to reformat the document. Simply select Code Format: Format Code from the command palette.

Formatting

Go to definition

 

Ctrl click on a class or variable reference to be taken directly to where it is defined within your project.

Hover

 

Hover the mouse pointer over a type or other supported object and you can see some additional information relating to it.

Outline view

 

Many providers let you see a tree-based outline view of the current document which you can search and then click to go right to that area of code. You can toggle the new Outline View from the View menu or the Outline View: Toggle command.

Reference highlighting

 

Some providers let you see immediate references to the variable or object you are working on. Positioning your text cursor within that variable can highlight other references instantly.

Future plans

 

This is just the start of our journey. With the help of our community, we plan to expand the number of languages that Atom-IDE can support and make it possible for you to run and edit applications, making Atom-IDE a true IDE.

We hope to see future language support for the great languages out there including Rust, Go, Python, etc.

If a language server exists for your favorite language it is incredibly easy to create your own Atom-IDE package that takes advantage of it by using our atom-languageclient npm library that provides common automatic wire-up of the major features as well as helper tools such as downloading support files and conversions.

 

GitHub project of the week: Yarn 1.0

The team behind Yarn, an open-source, fast and secure alternative npm client, announced the 1.0 release of the JavaScript package manager, which is a major step for the project. In the 11 month since its initial release in 2016 has generated more than 175,000 projects on GitHub, and it’s responsible for nearly three billion package downloads per month.

So what’s new in Yarn? Yarn added a new feature called Workspaces, which lets people automatically aggregate all the dependencies from multiple package.json files and install them all in one go. It also uses a single yarn.lock file at the root, to lock them all, according to a Facebook post debuting the Yarn 1.0 release.

Workspaces is used by some teams at Facebook already, like in Babel. Lerna, a mono-repository management tool lets you opt in to Yarn’s Workspaces.

“By making Workspaces native to Yarn, we hope to enable faster and lighter installations by preventing package duplication between the smaller parts of a larger project,” read the Facebook blog.
Also in Yarn 1.0 is the new auto-merging of lockfiles feature. When there’s a merge conflict in the lockfile, Yarn will automatically handle the conflict resolution for you upon running yarn install, according to the blog. And if it succeeds, the conflict-free lockfile will save to a disk.

The next time you have a lockfile conflict, you can save time by running yarn install instead of doing a manual resolution, according to the Yarn team.

Besides some of the top new features, Yarn also improved its interactive upgrade experience, it includes a faster file integrity check, and there’s a separate lockfile parser module that you can use in your project.

Top five projects trending on GitHub this week

#1. Every Programmer Should Know: A collection of (mostly) technical things every software developer should know

#2. R2: HTTP client. Spiritual successor to request.

#3. WTFPython: A collection of interesting, subtle, and tricky Python snippets.

#4. Easy Mock: A persistent service that generates mock data quickly and provides visualization view.

#5. Clean Code PHP:🛁 Clean Code concepts adapted for PHP

GitHub project of the week: Downshift

When Paypal realized they’d need a versatile item selection solution for their site, engineer Kent C. Dodds sat down and came up with Downshift.

Downshift is a set of primitives to build simple, flexible, WAI-ARIA compliant React autocomplete/dropdown/select/combobox components. It will be rolled out on Paypal.com next week, but in a post on Medium, Dodds outlined how his code, built with Facebook’s React JavaScript library, is a versatile outline for item selection in all its forms.

“We actually have several other item selection experiences in our app that have slightly different use cases and necessitated us having multiple implementations of an autocomplete component in the same app!” Dodds says in the post. “So that’s why I build downshift: so we could have a single implementation that was flexible enough to cover all the use cases.”

Dodds credits the versatility of his solution on his use of “Function as Child” and “Controlled Props” patterns, taking the rendering of menu selections out of React’s hands and leaving it up to the developer to determine what’s rendered in the selection field and how it’s rendered.

Also, Downshift has slowly been gaining popularity. It already has 900 stars on GitHub and 7,000 downloads per month, even before its official 1.0 release. That means it’s being used in several places, but according to Dodds, the first production deployment that he is aware of is in codesandbox from Ives van Hoorne.

Top 5 projects trending on GitHub this week:
#1. Interface: The Interface font family
#2. rendertron: A dockerized, headless Chrome rendering solution
#3. write-a-hash-table: Learn how to write a hash table in C
#4. weweChat: Unofficial WeChat client built with React, MobX and Electron.
#5. oni: An IDE powered by Neovim

Compact electric vehicles can be a great alternative for urban commuters concerned with crowded roads and carbon emissions. With Pacer, Faraday Motion wants to provide a set of standard, modular components paired with open-source software for developers and makers to create affordable and simple electric vehicles.

The “brain” behind the Berlin-based startup’s modular hardware is the NODEMCU ESP8266 WiFi module running their open-source software, which allows the chip to connect to a smartphone or WiFi enabled controller.

Riders can control their speed via their phone’s tilt sensor or the controller’s joystick.

According to their website, CEO Sune Pedersen was inspired to start Faraday Motion after an accident left him with permanent mobility problems in his knee. The Hyperboard electric skateboard was his first hardware design, having come from a software development background.

Though Faraday Motion’s Pacer and Hyperboard are both works-in-progress, they envision their designs and software being used to motorize other vehicles, like bicycles and wheelchairs.

Top 5 projects trending on GitHub this week:
#1. Puppeteer
: Node library which provides a high-level API to control headless Chrome over the DevTools Protocol
#2. EffectiveTensorflow: TensorFlow tutorials and best practices.
#3. learn-regex: Learn regex the easy way.
#4. gtop: System monitoring dashboard for terminal.
#5. Awesome-Hacking: A collection of various awesome lists for hackers, pentesters and security researchers.

Oracle continues to make progress Java EE 8, the enterprise edition for the Java platform, and moving forward it would like to advance Java EE within a more open and collaborative community. Specifications are nearly complete and the Java team expects to deliver the Java EE 8 reference implementation this summer.

As the delivery of Java EE 8 approaches, Oracle believes they have the ability to rethink how Java EE is developed in order to “make it more agile and responsive to changing industry and technology demands.”

“Java EE is enormously successful, with a competitive market of compatible implementations, broad adoption of individual technologies, a huge ecosystem of frameworks and tools, and countless applications delivering value to enterprises and end users,” according to Oracle in a blog post. “But although Java EE is developed in open source with the participation of the Java EE community, often the process is not seen as being agile, flexible or open enough, particularly when compared to other open source communities. We’d like to do better.”

According to Oracle, moving Java EE technologies to an open-source foundation may be the right next step, in order to “adopt more agile processes, implement more flexible licensing, and change the governance process.” Oracle also plans on exploring this possibility with the developer community, its licensees and several candidate foundations to see if they can move Java EE in this direction.

“We believe a more open process, that is not dependent on a single vendor as platform lead, will encourage greater participation and innovation, and will be in best interests of the community,” reads the blog.

While there are many details that need to be fleshed out, Red Hat’s John Clingan, senior principal product manager, said that Red Hat is optimistic and applauds Oracle’s decision to advance Java EE under an open-source foundation. Red Hat, an open-source software company, is built on the principles of the open-source way.

“We think that putting Java EE under the jurisdiction of an open source organization is a very positive move that will benefit the entire Enterprise Java community,” said Clingan.

Since Java EE has been evolving for nearly two decades to address market needs, Clingan said that Red Hat believes that a two-tier approach is needed to evolve Java EE more quickly.

“This includes Java EE as a standard, which should move at a measured pace, and Eclipse MicroProfile as an open-source project that acts as an innovation engine to drive new features for Java EE developers more quickly,” said Clingan. The Configuration JSR submission is an example, he added.

As an Eclipse MicroProfile community member, Red Hat plans to continue forward and deliver functional specifications within the Eclipse MicroProfile community as the effort to move Java EE to a foundation progresses. And as a licensee, Red Hat (and JBoss before its acquisition) pioneered the idea of an open-standard enterprise application platform and open-source collaboration, and according to Clingan, it really drove open-source adoption into the “heart of the enterprise,” he said.

Red Hat leads the CDI and Bean Validation Java EE-related JSRs, participates in multiple additional Java EE-related JSRs, and it ships JBoss Enterprise Application Platform as fully Java EE-compatible, said Clingan.

As Java EE moves forward, Oracle writes that it intends on meeting the needs of its developers, end users, customers, technology consumers, partners and licensees. Clingan said that Java EE has the opportunity to grow even more, and with a more permissive license, it will encourage new contributions, new implementations and distributions, he said. And, end-user developers should be able to use Java EE-related technologies more quickly.

Also, Oracle will support existing Java EE implementations and future implementations of Java EE.

The production release of the Oracle Database Programming Interface for C (ODPI-C), which gives more streamlined access to C and C++ developers to Oracle Database, has been launched on GitHub.

The open-source wrapper is aimed primarily at language interface developers, allowing users to quickly call more common features of the Oracle Call Interface (OCI), the main C API for Oracle Database. But the company says that its conciseness makes it a flexible and accessible tool.

The library is being used internally by Oracle for the Python cx_Oracle 6 interface and has already been implemented in Python, Node.js, Go and Rust interfaces and in custom applications.

ODPI-C aims to simplify memory and resource management when binding and defining data with a reference counting mechanism that stops applications from destroying resources that are in-use.

The project was an effort by the Oracle Database Data Access team, who maintain OCI and additional APIs for Oracle Database. ODPI-C was led by developer Anthony Tuininga, who also leads development of the cx_Oracle interface for Python.

Version 6 of cx_Oracle features support for the new ODPI-C abstraction layer. Notable changes include compatibility with Python Wheels and improvements to scalability, all made possible by ODPI-C.

The ODPI-C source code is available under the Apache 2.0 and Oracle UPL licenses for direct inclusion in the code base of an interface or other project.

It has been tested on Windows, Linux and OS X, with a minimum requirement of Visual Studio 2008, GCC 4.4 and Xcode 6, respectively.