How to Produce a Buffer Overflowed Program in Modern Visual Studio

12/04/2018

To study binary security, it is very important for learners to write a program with buffer overflow vulnerability in order to understand how buffer overflow works. However, modern IDEs or compilers usually have default settings which prevent such thing from happening easily.

To write, compile and run a simple program with buffer overflow vulnerability, we need to change some default setting. This article will use Microsoft Visual Studio Community 2017 Version 15.9.6 as an example to illustrate the steps needed to produce a vulnerable C program.

The codes of the C program is:

#include <stdio.h>
#include <string.h>

#define PASSWORD "secret233"
#define BUFFER_SIZE 10

int check_pass(char *input)
{
    int compare = 1;
    char buffer[BUFFER_SIZE];

    compare = strcmp(input, PASSWORD);
    printf("[matched value]:%d\n", compare);

    strcpy(buffer, input);
    printf("[matched value]:%d\n", compare);

    return !compare;
}

main()
{
    int passed = 0;
    char input[1024];

    while (1) {
        printf("Enter password: ");
        scanf("%s", input);

        passed = check_pass(input);
        if (passed) {
            printf("--Password correct!\n");
            break;
        }
        else
            printf("--Wrong password. Try again.\n\n");
    }
}

Create a C program project

The first step is to create a C program project. To do that, open Visual Studio and select File->New->Project. In the next window, choose “Visual C++” on the left side and Empty Project in the middle.

After the project is created, add a “main.c” file in the “Source Files” folder, put the codes above into the .c file. This will be the only source code file we need.

By default, Visual Studio 2017 compiles the codes with C++ standard. To change it to C, right-click the project name in Solution Explorer and select Properties to open project property page.

In the property configuration page, make sure the “Configuration” profile is set as “Debug”. Then in the C/C++ -> Advanced page, set Compile As option as Compile as C Code (/TC). This adds the /TC flag to compiler, forcing it to use C standard to compile our codes.

overflow

In modern Visual Studio, when debugging such a C/C++ program, a console window will pop up and close automatically when the program exits. To prevent the debug console from closing, go to Linker->System, set SubSystem option as Console (/SUBSYSTEM:CONSOLE).

Disable security warnings

Now try to build the project. Visual Studio will produce two errors saying ‘strcpy’ and ‘scanf’ functions are unsafe. (Indeed, they are the exact cause of buffer overflow vulnerability we want to make).

In order to compile the codes without error, open project properties page again. Go to C/C++ -> Preprocessor page, find the Preprocessor Definitions option at the top. Open the dropdown menu to the right and click Edit….

overflow

In the next window, enter _CRT_SECURE_NO_WARNINGS in the top text box, then click OK. This will add definition to preprocessor to disable security warnings.

overflow

Now build the project and use Ctrl+F5 to run the program. The current program works decently, it tests every user input and passes only when the input string matches pre-defined password.

overflow

If you enter a string that is longer than 9 characters, a runtime error will occur since buffer overflow is detected. It happens because the total size of the input string (characters + ending ‘\0’) exceeds the size of buffer when ‘strcpy’ is called.

overflow

Therefore, although we are very close to implementing a successful buffer overflow vulnerability, there’s still one more step to do.

Disable buffer overflow runtime check

Open project properties page one last time. This time, go to C/C++ -> Code Generation. Find the option Basic Runtime Checks and set it as Default or Uninitialized variables (/RTCu).

overflow

Why do we need to change this value? Because /RTC is the VC compiler option that controls the runtime error checks. Among different values of this option, /RTCs is the one that enables stack frame runtime error checking. Therefore, by switching to an option that doesn’t contain /RTCs, the buffer overflow won’t be detected once happens. For more details about this compiler option, read this article on Microsoft website.

Test

Now the program is all ready. Build and run the program with debug configuration. In the example below, a buffer overflow vulnerability was successfully exploited, making the input string passed the verification without using the correct password.

overflow

For those who don’t know much about buffer overflow, you may be wondering how this result was produced. Go search and learn more about buffer overflow, you will know why and have better understanding about how programs, memory and stack frame work.

For those who know buffer overflow, see if you can answer the following question: In this example, what kind of input string can pass the password verification by exploiting buffer overflow?

Think first, and then run the program by yourself to verify your answer!

SYANG.IO