Hadoop - Winutils

Yarn Hortonworks


Provide basic command line utilities for Hadoop on Windows.

It works with hadoop.dll to get local access to the local file system ?


winutils.exe --help
Usage: hadoop\bin\winutils.exe [command] ...
Provide basic command line utilities for Hadoop on Windows.

The available commands and their usages are:

chmod          Change file mode bits.

   or: chmod [OPTION] MODE [FILE]
Change the mode of the FILE to MODE.

   -R: change files and directories recursively

Each MODE is of the form '[ugoa]*([-+=]([rwxX]*|[ugo]))+'.

chown          Change file owner.

Usage: chown [OWNER][:[GROUP]] [FILE]
Change the owner and/or group of the FILE to OWNER and/or GROUP.

On Linux, if a colon but no group name follows the user name, the group of
the files is changed to that user's login group. Windows has no concept of
a user's login group. So we do not change the group owner in this case.

groups         List user groups.

Usage: groups [OPTIONS] [USERNAME]
Print group information of the specified USERNAME (the current user by default).

OPTIONS: -F format the output by separating tokens with a pipe

hardlink       Hard link operations.

Usage: hardlink create [LINKNAME] [FILENAME] |
       hardlink stat [FILENAME]
Creates a new hardlink on the existing file or displays the number of links
for the given file

ls             List file information.

Usage: ls [OPTIONS] [FILE]
List information about the FILE (the current directory by default).
Using long listing format and list directory entries instead of contents,
and do not dereference symbolic links.
Provides equivalent or similar function as 'ls -ld' on GNU/Linux.

OPTIONS: -L dereference symbolic links
         -F format the output by separating tokens with a pipe

symlink   Create a symbolic link.

Usage: symlink [LINKNAME] [FILENAME]
Creates a symbolic link

0 is returned on success.
2 is returned if the user does no have privilege to create symbolic links.
1 is returned for all other errors.

The default security settings in Windows disallow non-elevated administrators
and all non-administrators from creating symbolic links. The security settings
for symbolic links can be changed in the Local Security Policy management

readlink  Print the target of a symbolic link.

Usage: readlink [LINKNAME]
Prints the target of a symbolic link
The output and returned error codes are similar to the UNIX
readlink command. However no options are accepted.

0 is returned on success.
1 is returned for all errors.

systeminfo     System information.

    Usage: systeminfo
    Prints machine information on stdout
    Comma separated list of the following values.

task           Task operations.

    Usage: task create [TASKNAME] [COMMAND_LINE] |
          task createAsUser [TASKNAME] [USERNAME] [PIDFILE] [COMMAND_LINE] |
          task isAlive [TASKNAME] |
          task kill [TASKNAME]
          task processList [TASKNAME]
    Creates a new task jobobject with taskname
    Creates a new task jobobject with taskname as the user provided
    Checks if task jobobject is alive
    Kills task jobobject
    Prints to stdout a list of processes in the task
    along with their resource usage. One process per line
    and comma separated info per process
service        Service operations.

    Usage: service
    Starts the nodemanager Windows Secure Container Executor helper service.
    The service must run as a high privileged account (LocalSystem)
    and is used by the nodemanager WSCE to spawn secure containers on Windows.


Download and installation

You can download a compiled version at https://github.com/steveloughran/winutils/ and put it in the HADOOP_HOME\bin.

RStudio comes also with on at RStudio_home\bin\winutils

# Example for hadoop 2.7


Could not locate executable null\bin\winutils.exe

2018-06-04 20:23:33 ERROR Shell:397 - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:387)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)

Have you specified HADOOP_HOME ?

FindFileOwnerAndPermission error (1789): The trust relationship between this workstation and the primary domain failed.

Winutils tries to authenticate you against your domain and fails.

Unfortuantely, if this a corporate computer, you need to be connected to your corporate network to be able to authenticate against your domain.

Discover More
Yarn Hortonworks
Hadoop - Cluster Installation

Manual installation of an hadoop cluster. To start a Hadoop cluster you will need to start both the HDFS and YARN cluster. More ......
Card Puncher Data Processing
Hive - Scratch Directory

Scratch space for Hive jobs. This directory is used by Hive to store: the plans for different map/reduce stages for the query intermediate outputs of plan operations The default value is...
Idea Classpath Spark
Spark - Local Installation

A local installation is a spark installation on a single machine (generally a dev machine). The local master connection will start for you a local standalone spark installation on your machine. This...
Idea Classpath Spark
Spark Java - Installation and configuration on IDEA

How to configure IDEA against Spark with Java This section shows the important configuration that you need to pass to any Spark App in order to have a valid run. The HADOOP_HOME environment variable...

Share this page:
Follow us:
Task Runner